深海游弋的鱼 – 默默的点滴

ubuntu 22.04.2编译Android 12.1源代码&模拟器

Android 11系统映像可直接将 ARM 指令转换成 x86 指令，因此以前很多需要真机测试的情况，现在只需要模拟器就可以进行操作了。

不过，根据官方博客 Run ARM apps on the Android Emulator 尾部的一段注意事项：

这段事项说明，自己编译的镜像是没办法用上这个功能的，必须是Google编译好的镜像。

由于众所周知的原因，我们是没办法正常下载Android的源代码的，因此只能使用国内的镜像来操作了。

1.安装repo工具以及依赖

$ sudo apt-get install gcc make perl

$ sudo apt-get install ccache

$ sudo apt-get install curl

$ sudo apt-get install python3

# 升级pip
$ pip3 install pip --upgrade

# 清理可能的无效缓存
$ pip cache purge

# 安装repo
$ sudo apt-get install repo

$ sudo apt-get install aria2

$ sudo apt-get install git

$ sudo apt install -y libssl-dev

# 解决 
# prebuilts/clang/host/linux-x86/clang-3289846/bin/clang.real: error while loadin
g shared libraries: libncurses.so.5: cannot open shared object file: No such fi
le or directory

$ sudo apt install libncurses5


# 安装VSCode，方便后续查看代码
$ sudo snap install code --classic

$ git config --global user.email "user@email.com"

$ git config --global user.name "user"

$ sudo apt-get install gcc make perl

$ sudo apt-get install ccache

$ sudo apt-get install curl

$ sudo apt-get install python3

# 升级pip

$ pip3 install pip --upgrade

# 清理可能的无效缓存

$ pip cache purge

# 安装repo

$ sudo apt-get install repo

$ sudo apt-get install aria2

$ sudo apt-get install git

$ sudo apt install -y libssl-dev

# 解决

# prebuilts/clang/host/linux-x86/clang-3289846/bin/clang.real: error while loadin

g shared libraries: libncurses.so.5: cannot open shared object file: No such fi

le or directory

$ sudo apt install libncurses5

# 安装VSCode，方便后续查看代码

$ sudo snap install code --classic

$ git config --global user.email "user@email.com"

$ git config --global user.name "user"

2.在需要存储代码的地方创建文件夹

$ mkdir ~/AndSrc

$ cd ~/AndSrc

$ mkdir ~/AndSrc

$ cd ~/AndSrc

3.使用镜像下载Android源代码

清华大学的镜像

# 针对ubuntu 22.04不建议直接下载压缩包，建议直接使用repo更新，原因是ubuntu 22.04的python被升级成了python3，python2被完全移除。
# 而清华镜像压缩包里面的repo还是python2时候创建的，会引起各种初始化异常
# aria2 -c https://mirrors.ustc.edu.cn/aosp-monthly/aosp-latest.tar

$ mkdir aosp

$ cd aosp

$ repo init -u https://aosp.tuna.tsinghua.edu.cn/platform/manifest --repo-url=https://gerrit-googlesource.lug.ustc.edu.cn/git-repo

# 针对ubuntu 22.04不建议直接下载压缩包，建议直接使用repo更新，原因是ubuntu 22.04的python被升级成了python3，python2被完全移除。

# 而清华镜像压缩包里面的repo还是python2时候创建的，会引起各种初始化异常

# aria2 -c https://mirrors.ustc.edu.cn/aosp-monthly/aosp-latest.tar

$ mkdir aosp

$ cd aosp

$ repo init -u https://aosp.tuna.tsinghua.edu.cn/platform/manifest --repo-url=https://gerrit-googlesource.lug.ustc.edu.cn/git-repo

4.Android 模拟器编译(可选）

# Android 12模拟器对应分支 emu-31-release 

$ repo init -b emu-31-release 

# 会失败多次，不断尝试，直到完全成功,目前测试发现单线程虽然慢点，但是基本不会失败，多线程经常失败
$ repo sync -j4 --fail-fast

$ cd external/qemu/android/

$ ./rebuild.sh --no-tests

# Android 12模拟器对应分支 emu-31-release

$ repo init -b emu-31-release

# 会失败多次，不断尝试，直到完全成功,目前测试发现单线程虽然慢点，但是基本不会失败，多线程经常失败

$ repo sync -j4 --fail-fast

$ cd external/qemu/android/

$ ./rebuild.sh --no-tests

编译完成之后，产生的模拟器可执行文件及库文件都位于external/qemu/objs/android目录下：

~/AndSrc/aosp/external/qemu/android/objs$ ls
android_emu64_unittests           emulator64-mips
android_emu_metrics64_unittests   emulator64_simg2img
bin64                             emulator64_test_crasher
build                             emulator64-x86
emugl64_common_host_unittests     emulator-check
emulator                          lib
emulator64-arm                    lib64
emulator64_crashreport_unittests  lib64GLcommon_unittests
emulator64-crash-service          lib64OpenglRender_unittests
emulator64_img2simg               qemu
emulator64_libui_unittests        resources
emulator64_make_ext4fs

~/AndSrc/aosp/external/qemu/android/objs$ ls

android_emu64_unittests emulator64-mips

android_emu_metrics64_unittests emulator64_simg2img

bin64 emulator64_test_crasher

build emulator64-x86

emugl64_common_host_unittests emulator-check

emulator lib

emulator64-arm lib64

emulator64_crashreport_unittests lib64GLcommon_unittests

emulator64-crash-service lib64OpenglRender_unittests

emulator64_img2simg qemu

emulator64_libui_unittests resources

emulator64_make_ext4fs

后面就可以像执行 SDK 中的模拟器那样，执行我们编译的模拟器了：

# 下面是个例子，只有设备上已经通过Android SDK 创建过 Nexus_5_API_30_x86，才可以成功执行

~/AndSrc/aosp/external/qemu/android/objs$ ./emulator -avd Nexus_5_API_30_x86

# 下面是个例子，只有设备上已经通过Android SDK 创建过 Nexus_5_API_30_x86，才可以成功执行

~/AndSrc/aosp/external/qemu/android/objs$ ./emulator -avd Nexus_5_API_30_x86

5.列出android-12全部分支

$ cd ~/AndSrc/aosp

$ cd .repo/manifests && git branch -a | cut -d / -f 3 | grep android-12

$ cd ~/AndSrc/aosp

$ cd .repo/manifests && git branch -a | cut -d / -f 3 | grep android-12

6.编译Android 12系统镜像

$ cd ~/AndSrc/aosp

# 前面的编译可能会产生部分垃圾文件，需要清理一下,清理之前，编译好的文件提前备份一下	 
$ rm -rf *

$ repo init -b android-12.1.0_r26

# 会失败多次，不断尝试，直到完全成功
$ repo sync -j4 --fail-fast

# 如果上面的命令失败，可以尝试调整为单线程,目前测试发现单线程虽然慢点，但是基本不会失败，多线程经常失败 
# repo sync -j1 --fail-fast

$ cd ~/AndSrc/aosp

# 前面的编译可能会产生部分垃圾文件，需要清理一下,清理之前，编译好的文件提前备份一下

$ rm -rf *

$ repo init -b android-12.1.0_r26

# 会失败多次，不断尝试，直到完全成功

$ repo sync -j4 --fail-fast

# 如果上面的命令失败，可以尝试调整为单线程,目前测试发现单线程虽然慢点，但是基本不会失败，多线程经常失败

# repo sync -j1 --fail-fast

7.引入编译环境变量

$ source build/envsetup.sh

1	$ source build/envsetup.sh

8.设置编译目标，此处我们指定编译x86_64下的完整调试版本（镜像无法安装ARM应用）

# 纯x86系统镜像，不能安装arm应用
# lunch sdk_phone_x86_64

# eng：代表 engineer，开发工程师的版本，拥有最大的权限(root等)，具有额外调试工具的开发配置。 
# 执行 lunch 命令可以输出全部的编译目标列表 
# 更多的编译选项可以从 build/make/target/product/ 下看到 
# lunch aosp_x86_arm-eng 编译后的镜像可以安装 arm 应用, 不过在不进行任何代码调整的情况下，应用启动会崩溃 
# 不能安装 arm 应用

$ lunch sdk_phone_x86_64-eng

# 纯x86系统镜像，不能安装arm应用

# lunch sdk_phone_x86_64

# eng：代表 engineer，开发工程师的版本，拥有最大的权限(root等)，具有额外调试工具的开发配置。

# 执行 lunch 命令可以输出全部的编译目标列表

# 更多的编译选项可以从 build/make/target/product/ 下看到

# lunch aosp_x86_arm-eng 编译后的镜像可以安装 arm 应用, 不过在不进行任何代码调整的情况下，应用启动会崩溃

# 不能安装 arm 应用

$ lunch sdk_phone_x86_64-eng

9.如果需要跟踪调试代码，建议编译为调试类型

$ export TARGET_BUILD_TYPE=debug

1	$ export TARGET_BUILD_TYPE=debug

10.编译

$ export USE_CCACHE=1  

# 清理代码，为编译准备干净的环境

$ make clobber

# 由于编译的时候内存压力会非常大，即使我们已经配置了足够大的交换分区
# 如果系统内存在16GB左右，非常容易触发 ubuntu 22.04 自带的 systemd-oomd 杀进程操作
# 导致编译过程莫名中断,如果发现编译的时候，命令行窗口突然消失了，一般都是 systemd-oomd引起的
# 此时我们需要暂时停止 systemd-oomd
# sudo systemctl stop systemd-oomd

$ make -j8

$ export USE_CCACHE=1

# 清理代码，为编译准备干净的环境

$ make clobber

# 由于编译的时候内存压力会非常大，即使我们已经配置了足够大的交换分区

# 如果系统内存在16GB左右，非常容易触发 ubuntu 22.04 自带的 systemd-oomd 杀进程操作

# 导致编译过程莫名中断,如果发现编译的时候，命令行窗口突然消失了，一般都是 systemd-oomd引起的

# 此时我们需要暂时停止 systemd-oomd

# sudo systemctl stop systemd-oomd

$ make -j8

注意此处如果发生编译失败，原因基本上是编译顺序导致的引用出错，也就是某些模块还没有编译完成，其他模块已经开始尝试链接，导致依赖错误，此时只要把多线程并发编译修改成单线程编译即可，即直接执行

$ make

$ make

如果由于内存不足导致的编译失败，可以增加物理内存。但是如果内存无法增加的话，那么适当增加交换分区/交换文件的大小（建议配置16GB以上的交换分区）可以解决此问题，只是编译速度会下降。

运行镜像

选择system-qemu.img和vendor-qemu.img，这两个镜像是专门为qemu运行制作的，如果选择system.img 和vendor.img，则avd运行失败。

$ cd ~/AndSrc/aosp

$ export ANDROID_BUILD_TOP=~/AndSrc/aosp

$ export PATH=$PATH:$ANDROID_BUILD_TOP/out/host/linux-x86/bin

$ export ANDROID_SWT=$ANDROID_BUILD_TOP/out/host/linux-x86/framework

$ export ANDROID_PRODUCT_OUT=$ANDROID_BUILD_TOP/out/target/product/emulator_x86_64

$ ./prebuilts/android-emulator/linux-x86_64/emulator -verbose -show-kernel

$ cd ~/AndSrc/aosp

$ export ANDROID_BUILD_TOP=~/AndSrc/aosp

$ export PATH=$PATH:$ANDROID_BUILD_TOP/out/host/linux-x86/bin

$ export ANDROID_SWT=$ANDROID_BUILD_TOP/out/host/linux-x86/framework

$ export ANDROID_PRODUCT_OUT=$ANDROID_BUILD_TOP/out/target/product/emulator_x86_64

$ ./prebuilts/android-emulator/linux-x86_64/emulator -verbose -show-kernel

上面运行起来的镜像是从~/AndSrc/aosp/out/debug/target/product/generic/hardware-qemu.ini即可读取配置信息的，但是这个文件直接修改无效，我们如果需要修改参数，只能从启动参数中设置。
比如我们如果需要增大内存，开启GPU的支持，则执行如下命令：

$ ./prebuilts/android-emulator/linux-x86_64/emulator -gpu on -memory 4096 -verbose -show-kernel

1	$ ./prebuilts/android-emulator/linux-x86_64/emulator -gpu on -memory 4096 -verbose -show-kernel

编译支持ARM应用的镜像

尽管根据

我们自己编译的镜像是没办法直接从源代码编译支持安装运行 ARM 应用的。

但是有两个变通方案:

从Google官方的 Android 12 镜像中提取需要的文件塞到我们自己编译的镜像里，参考方案： Adding ARM native bridge to the AOSP11 x86 emulator、android_vendor_google_emu-x86
使用 Intel Houdini 实现支持，参考方案：Include Intel Houdini in Android-x86

参考链接

macOS TimeMachine恢复隐藏文件

macOS自带的时间机器（TimeMachine）还是很好用的，只是如果误删的文件是隐藏文件，操作起来稍微复杂了一些。具体参考如下：

Open the Terminal.
Run the following command:

$ defaults write com.apple.finder AppleShowAllFiles TRUE;killall Finder

1

$ defaults write com.apple.finder AppleShowAllFiles TRUE;killall Finder
Go to the location of the missing hidden folder and invoke Time Machine to restore it.
Run the following command to hide files:

$ defaults write com.apple.finder AppleShowAllFiles FALSE;killall Finder

1

$ defaults write com.apple.finder AppleShowAllFiles FALSE;killall Finder

参考链接

How to access hidden files to restore in Time Machine

Xcode 12.x添加iOS 10.x模拟器

背景

笔者昨天遇到个问题，有用户反馈在iOS 10.3.1的手机上，有个网页打开白屏。但是问题是笔者手头都没有10.x版本的手机，所以想安装模拟器来尝试复现。

然而，笔者发现电脑上的Xcode版本是12.5.1，已经不支持iOS 10.3.1的模拟器下载了。

继续阅读Xcode 12.x添加iOS 10.x模拟器

MMKV编译报错Invalid `Podfile` file: undefined method `exists?' for File:Class.

尝试在 Flutter 上使用 MMKV （1.2.16）的时候，编译报错，如下：

Launching lib/main.dart on iPod touch (7th generation) in debug mode...
Updating minimum iOS deployment target to 11.0.
Upgrading Podfile
Running pod install...
CocoaPods' output:
↳

    [!] Invalid `Podfile` file: undefined method `exists?' for File:Class.

     #  from /Users/xxxx/MMKV/flutter/example/ios/Podfile:35
     #  -------------------------------------------
     #    plugin_deps_file = File.expand_path(File.join(flutter_application_path, '..', '.flutter-plugins-dependencies'))
     >    if not File.exists?(plugin_deps_file)
     #      is_module = true;
     #  -------------------------------------------

    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:335:in `rescue in block in from_ruby'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:329:in `block in from_ruby'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:50:in `instance_eval'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:50:in `initialize'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:327:in `new'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:327:in `from_ruby'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:293:in `from_file'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-1.12.0/lib/cocoapods/config.rb:205:in `podfile'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-1.12.0/lib/cocoapods/command.rb:160:in `verify_podfile_exists!'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-1.12.0/lib/cocoapods/command/install.rb:46:in `run'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/claide-1.1.0/lib/claide/command.rb:334:in `run'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-1.12.0/lib/cocoapods/command.rb:52:in `run'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-1.12.0/bin/pod:55:in `<top (required)>'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/bin/pod:25:in `load'
    /usr/local/Cellar/cocoapods/1.12.0/libexec/bin/pod:25:in `<main>'

Error running pod install
Error launching application on iPod touch (7th generation).

Launching lib/main.dart on iPod touch (7th generation) in debug mode...

Updating minimum iOS deployment target to 11.0.

Upgrading Podfile

Running pod install...

CocoaPods' output:

↳

[!] Invalid `Podfile` file: undefined method `exists?' for File:Class.

# from /Users/xxxx/MMKV/flutter/example/ios/Podfile:35

# -------------------------------------------

# plugin_deps_file = File.expand_path(File.join(flutter_application_path, '..', '.flutter-plugins-dependencies'))

> if not File.exists?(plugin_deps_file)

# is_module = true;

# -------------------------------------------

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:335:in `rescue in block in from_ruby'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:329:in `block in from_ruby'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:50:in `instance_eval'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:50:in `initialize'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:327:in `new'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:327:in `from_ruby'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-core-1.12.0/lib/cocoapods-core/podfile.rb:293:in `from_file'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-1.12.0/lib/cocoapods/config.rb:205:in `podfile'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-1.12.0/lib/cocoapods/command.rb:160:in `verify_podfile_exists!'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-1.12.0/lib/cocoapods/command/install.rb:46:in `run'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/claide-1.1.0/lib/claide/command.rb:334:in `run'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-1.12.0/lib/cocoapods/command.rb:52:in `run'

/usr/local/Cellar/cocoapods/1.12.0/libexec/gems/cocoapods-1.12.0/bin/pod:55:in `<top (required)>'

/usr/local/Cellar/cocoapods/1.12.0/libexec/bin/pod:25:in `load'

/usr/local/Cellar/cocoapods/1.12.0/libexec/bin/pod:25:in `<main>'

Error running pod install

Error launching application on iPod touch (7th generation).

$ ruby -v
ruby 2.6.10p210 (2022-04-12 revision 67958) [universal.x86_64-darwin20]

1 2	$ ruby -v ruby 2.6.10p210 (2022-04-12 revision 67958) [universal.x86_64-darwin20]

$ brew install ruby
==> Downloading https://formulae.brew.sh/api/formula.jws.json
######################################################################## 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json
######################################################################## 100.0%
Warning: ruby 3.2.1 is already installed and up-to-date.
To reinstall 3.2.1, run:
  brew reinstall ruby

$ ruby -v          
ruby 2.6.10p210 (2022-04-12 revision 67958) [universal.x86_64-darwin20]

$ brew link ruby   
Warning: Refusing to link macOS provided/shadowed software: ruby
If you need to have ruby first in your PATH, run:
  echo 'export PATH="/usr/local/opt/ruby/bin:$PATH"' >> ~/.zshrc

For compilers to find ruby you may need to set:
  export LDFLAGS="-L/usr/local/opt/ruby/lib"
  export CPPFLAGS="-I/usr/local/opt/ruby/include"

For pkg-config to find ruby you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/ruby/lib/pkgconfig"

$ brew install ruby

==> Downloading https://formulae.brew.sh/api/formula.jws.json

######################################################################## 100.0%

==> Downloading https://formulae.brew.sh/api/cask.jws.json

######################################################################## 100.0%

Warning: ruby 3.2.1 is already installed and up-to-date.

To reinstall 3.2.1, run:

brew reinstall ruby

$ ruby -v

ruby 2.6.10p210 (2022-04-12 revision 67958) [universal.x86_64-darwin20]

$ brew link ruby

Warning: Refusing to link macOS provided/shadowed software: ruby

If you need to have ruby first in your PATH, run:

echo 'export PATH="/usr/local/opt/ruby/bin:$PATH"' >> ~/.zshrc

For compilers to find ruby you may need to set:

export LDFLAGS="-L/usr/local/opt/ruby/lib"

export CPPFLAGS="-I/usr/local/opt/ruby/include"

For pkg-config to find ruby you may need to set:

export PKG_CONFIG_PATH="/usr/local/opt/ruby/lib/pkgconfig"

$ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.7.7, on macOS 11.7.4 20G1120 darwin-x64, locale zh-Hans-CN)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.1)
[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.1)
[✓] IntelliJ IDEA Ultimate Edition (version 2018.3.6)
[✓] VS Code (version 1.76.2)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

• No issues found!

$ flutter doctor

Doctor summary (to see all details, run flutter doctor -v):

[✓] Flutter (Channel stable, 3.7.7, on macOS 11.7.4 20G1120 darwin-x64, locale zh-Hans-CN)

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.1)

[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)

[✓] Chrome - develop for the web

[✓] Android Studio (version 2022.1)

[✓] IntelliJ IDEA Ultimate Edition (version 2018.3.6)

[✓] VS Code (version 1.76.2)

[✓] Connected device (3 available)

[✓] HTTP Host Availability

• No issues found!

这个问题是由于 cocoapods 升级到 1.12 版本之后，依赖的 ruby 升级到 3.2 版本，其中的 File.exists 函数被替换成 File.exist，导致编译异常。

刚刚开始以为是 Flutter 的原因，结果发现 Flutter 3.7.7版本已经修复这个问题。

尝试了半天，才发现是 MMKV 的问题，主要是 MMKV的 iOS 工程下 flutter/example/ios/Podfile 跟 flutter/example/mmkvpodhelper.rb里面的代码需要进行适配。

# from ruby 3.2  File.exists is broken, we need compat function
def mmkv_file_exists(file)
  is_exist = false
  if File.methods.include?(:exists?)
    is_exist = File.exists? file
  else
    is_exist = File.exist? file
  end
  return is_exist
end

# from ruby 3.2 File.exists is broken, we need compat function

def mmkv_file_exists(file)

is_exist = false

if File.methods.include?(:exists?)

is_exist = File.exists? file

else

is_exist = File.exist? file

end

return is_exist

end

使用上面的代码替换 File.exists 即可。

参考链接

关于HTTP请求走私的小记

0x00 写在前面

之前一次线上赛，遇到一道Web题，涉及了HTTP请求走私。由于之前未学习过，从而我展开了HTTP请求走私的学习之旅。

0x01 HTTP请求走私是什么

HTTP请求走私是一种干扰网站处理从一个或多个用户接收的HTTP请求序列的方式的技术。使攻击者可以绕过安全控制，未经授权访问敏感数据并直接危害其他应用程序用户。

0x02 为什么会产生HTTP请求走私

请求走私漏洞成因

前端服务器(CDN)和后端服务器接收数据不同步，引起对客户端传入的数据理解不一致，从而导致漏洞的产生。

大多数HTTP请求走私漏洞的出现是因为HTTP规范提供了两种不同的方法来指定请求的结束位置：Content-Length标头和Transfer-Encoding标头。
同时使用两种不同的方法时，Content-Length无效。当使用多个服务器时，对客户端传入的数据理解不一致时，就会出现有些服务器认为Content-Length的长度有效，有些以Transfer-Encoding有效。而一般情况下，反向代理服务器与后端的源站服务器之间，会重用TCP链接。这样超出的长度就会拼接到下一次请求进行请求，从而导致HTTP请求走私漏洞。

RFC2616规范

如果接收的消息同时包含传输编码头字段(Transfer-Encoding)和内容长度头(Content-Length)字段，则必须忽略后者。

由于规范默许可以使用Transfer-Encoding和Content-Length处理请求，因此很少有服务器拒绝此类请求。每当我们找到一种方法，将Transfer-Encoding隐藏在服务端的一个chain中时，它将会回退到使用Content-Length去发送请求。

走私攻击实现

当向代理服务器发送一个比较模糊的HTTP请求时，由于两者服务器的实现方式不同，代理服务器可能认为这是一个HTTP请求，然后将其转发给了后端的源站服务器，但源站服务器经过解析处理后，只认为其中的一部分为正常请求，剩下的那一部分，就算是走私的请求，当该部分对正常用户的请求造成了影响之后，就实现了HTTP走私攻击。

扩展：为什么会出现多次请求

这与最为广泛的HTTP 1.1的协议特性——Keep-Alive&Pipeline有关。

在HTTP1.0之前的协议设计中，客户端每进行一次HTTP请求，需要同服务器建立一个TCP链接。

而现代的Web页面是由多种资源组成的，要获取一个网页的内容，不仅要请求HTML文档，还有JS、CSS、图片等各种资源，如果按照之前的协议设计，就会导致HTTP服务器的负载开销增大。于是在HTTP1.1中，增加了Keep-Alive和Pipeline这两个特性。

Keep-Alive：在HTTP请求中增加一个特殊的请求头Connection: Keep-Alive，告诉服务器，接收完这次HTTP请求后，不要关闭TCP链接，后面对相同目标服务器的HTTP请求，重用这一个TCP链接。这样只需要进行一次TCP握手的过程，可以减少服务器的开销，节约资源，还能加快访问速度。这个特性在HTTP1.1中默认开启的。

Pipeline(http管线化)：http管线化是一项实现了多个http请求但不需要等待响应就能够写进同一个socket的技术，仅有http1.1规范支持http管线化。在这里，客户端可以像流水线一样发送自己的HTTP请求，而不需要等待服务器的响应，服务器那边接收到请求后，需要遵循先入先出机制，将请求和响应严格对应起来，再将响应发送给客户端。

现在，浏览器默认不启用Pipeline的，但是一般的服务器都提供了对Pipleline的支持。

继续阅读关于HTTP请求走私的小记

Transfer-Encoding 的作用

通过HTTP传送数据时，有些时候并不能事先确定body的长度，因此无法得到Content-Length的值，就不能在header中指定Content-Length了，造成的最直接的影响就是：接收方无法通过Content-Length得到报文体的长度，那怎么判断发送方发送完毕了呢？HTTP 1.1协议在header中引入了Transfer-Encoding，当其值为chunked时, 表明采用chunked编码方式来进行报文体的传输

HTTP 1.1中有两个实体头(Entity-Header)直接与编码相关,分别为Content-Encoding和Transfer-Encoding.
先说Content-Encoding, 该头表示实体已经采用了的编码方式.Content-Encoding是请求URL对应实体(Entity)本身的一部分.比如请求URL为http://host/image.png.gz时,可能会得到的Content-Encoding为gzip.Content-Encoding的值是不区分大小写的,目前HTTP1.1标准中已包括的有gzip/compress/deflate/identity等.
与Content-Encoding头对应,HTTP请求中包含了一个Accept-Encoding头,该头用来说明用户代理(User-Agent,一般也就是浏览器)能接受哪些类型的编码. 如果HTTP请求中不存在该头,服务器可以认为用户代理能接受任何编码类型.

接下来重点描述Transfer-Encoding, 该头表示为了达到安全传输或者数据压缩等目的而对实体进行的编码. Transfer-Encoding与Content-Encoding的不同之处在于:
1, Transfer-Encoding只是在传输过程中才有的,并非请求URL对应实体的本身特性.
2, Transfer-Encoding是一个"跳到跳"头,而Content-Encoding是"端到端"头.
该头的用途举例如,请求URL为http://host/abc.txt,服务器发送数据时认为该文件可用gzip方式压缩以节省带宽,接收端看到Transfer-Encoding为gzip首先进行解码然后才能得到请求实体.
此外多个编码可能同时对同一实体使用,所以Transfer-Encoding头中编码顺序相当重要,它代表了解码的顺序过程.同样,Transfer-Encoding的值也是不区分大小写的,目前HTTP1.1标准中已包括的有gzip/compress/deflate/identity/chunked等.
Transfer-Encoding中有一类特定编码:chunked编码.该编码将实体分块传送并逐块标明长度,直到长度为0块表示传输结束, 这在实体长度未知时特别有用(比如由数据库动态产生的数据). HTTP1.1标准规定,只要使用了Transfer-Encoding的地方就必须使用chunked编码,并且chunked必须为最后一层编码.任何HTTP 1.1应用都必须能处理chunked编码.
与Transfer-Encoding对应的请求头为TE,它主要表示请求发起者愿意接收的Transfer-Encoding类型. 如果TE为空或者不存在,则表示唯一能接受的类型为chunked.
其他与Transfer-Encoding相关的头还包括Trailer,它与chunked编码相关,就不细述了.

顾名思义,Content-Length表示传输的实体长度,以字节为单位(在请求方法为HEAD时表示会要发送的长度,但并不实际发送.).Content-Length受Transfer-Encoding影响很大,只要Transfer-Encoding不为identity,则实际传输长度由编码中的chunked决定,Content-Length即使存在也被忽略.

关于HTTP Message Body的长度
在HTTP中有消息体(Message body)和实体(Entity body)之分,简单说来在没有Transfer-Encoding作用时,消息体就是实体,而应用了Transfer-Encoding后,消息体就是编码后的实体,如下:

    Message body = Transfer-Encoding encode(Entity body)
如何确定消息体的长度? HTTP 1.1标准给出了如下方法(按照优先级依次排列):
    1, 响应状态(Response Status)为1xx/204/304或者请求方法为HEAD时,消息体长度为0.
    2, 如果使用了非"identity"的Transfer-Encoding编码方式,则消息体长度由"chunked"编码决定,除非该消息以连接关闭为结束.
    3, 如果存在"Content-Length"实体头,则消息长度为该数值.
    3, 如果消息使用关闭连接方式代表消息体结束,则长度由关闭前收到的长度决定. 该条对HTTP Request包含的消息体不适用.

Message body = Transfer-Encoding encode(Entity body)

如何确定消息体的长度? HTTP 1.1标准给出了如下方法(按照优先级依次排列):

1, 响应状态(Response Status)为1xx/204/304或者请求方法为HEAD时,消息体长度为0.

2, 如果使用了非"identity"的Transfer-Encoding编码方式,则消息体长度由"chunked"编码决定,除非该消息以连接关闭为结束.

3, 如果存在"Content-Length"实体头,则消息长度为该数值.

3, 如果消息使用关闭连接方式代表消息体结束,则长度由关闭前收到的长度决定. 该条对HTTP Request包含的消息体不适用.

具体详细的 RFC 7230 说明如下：

3.3.3.  Message Body Length

   The length of a message body is determined by one of the following
   (in order of precedence):

   1.  Any response to a HEAD request and any response with a 1xx
       (Informational), 204 (No Content), or 304 (Not Modified) status
       code is always terminated by the first empty line after the
       header fields, regardless of the header fields present in the
       message, and thus cannot contain a message body.

   2.  Any 2xx (Successful) response to a CONNECT request implies that
       the connection will become a tunnel immediately after the empty
       line that concludes the header fields.  A client MUST ignore any
       Content-Length or Transfer-Encoding header fields received in
       such a message.

   3.  If a Transfer-Encoding header field is present and the chunked
       transfer coding (Section 4.1) is the final encoding, the message
       body length is determined by reading and decoding the chunked
       data until the transfer coding indicates the data is complete.

       If a Transfer-Encoding header field is present in a response and
       the chunked transfer coding is not the final encoding, the
       message body length is determined by reading the connection until
       it is closed by the server.  If a Transfer-Encoding header field
       is present in a request and the chunked transfer coding is not
       the final encoding, the message body length cannot be determined
       reliably; the server MUST respond with the 400 (Bad Request)
       status code and then close the connection.

       If a message is received with both a Transfer-Encoding and a
       Content-Length header field, the Transfer-Encoding overrides the
       Content-Length.  Such a message might indicate an attempt to
       perform request smuggling (Section 9.5) or response splitting
       (Section 9.4) and ought to be handled as an error.  A sender MUST
       remove the received Content-Length field prior to forwarding such
       a message downstream.

   4.  If a message is received without Transfer-Encoding and with
       either multiple Content-Length header fields having differing
       field-values or a single Content-Length header field having an
       invalid value, then the message framing is invalid and the
       recipient MUST treat it as an unrecoverable error.  If this is a
       request message, the server MUST respond with a 400 (Bad Request)
       status code and then close the connection.  If this is a response
       message received by a proxy, the proxy MUST close the connection
       to the server, discard the received response, and send a 502 (Bad
       Gateway) response to the client.  If this is a response message
       received by a user agent, the user agent MUST close the
       connection to the server and discard the received response.

   5.  If a valid Content-Length header field is present without
       Transfer-Encoding, its decimal value defines the expected message
       body length in octets.  If the sender closes the connection or
       the recipient times out before the indicated number of octets are
       received, the recipient MUST consider the message to be
       incomplete and close the connection.

   6.  If this is a request message and none of the above are true, then
       the message body length is zero (no message body is present).

   7.  Otherwise, this is a response message without a declared message
       body length, so the message body length is determined by the
       number of octets received prior to the server closing the
       connection.

3.3.3. Message Body Length

The length of a message body is determined by one of the following

(in order of precedence):

1. Any response to a HEAD request and any response with a 1xx

(Informational), 204 (No Content), or 304 (Not Modified) status

code is always terminated by the first empty line after the

header fields, regardless of the header fields present in the

message, and thus cannot contain a message body.

2. Any 2xx (Successful) response to a CONNECT request implies that

the connection will become a tunnel immediately after the empty

line that concludes the header fields. A client MUST ignore any

Content-Length or Transfer-Encoding header fields received in

such a message.

3. If a Transfer-Encoding header field is present and the chunked

transfer coding (Section 4.1) is the final encoding, the message

body length is determined by reading and decoding the chunked

data until the transfer coding indicates the data is complete.

If a Transfer-Encoding header field is present in a response and

the chunked transfer coding is not the final encoding, the

message body length is determined by reading the connection until

it is closed by the server. If a Transfer-Encoding header field

is present in a request and the chunked transfer coding is not

the final encoding, the message body length cannot be determined

reliably; the server MUST respond with the 400 (Bad Request)

status code and then close the connection.

If a message is received with both a Transfer-Encoding and a

Content-Length header field, the Transfer-Encoding overrides the

Content-Length. Such a message might indicate an attempt to

perform request smuggling (Section 9.5) or response splitting

(Section 9.4) and ought to be handled as an error. A sender MUST

remove the received Content-Length field prior to forwarding such

a message downstream.

4. If a message is received without Transfer-Encoding and with

either multiple Content-Length header fields having differing

field-values or a single Content-Length header field having an

invalid value, then the message framing is invalid and the

recipient MUST treat it as an unrecoverable error. If this is a

request message, the server MUST respond with a 400 (Bad Request)

status code and then close the connection. If this is a response

message received by a proxy, the proxy MUST close the connection

to the server, discard the received response, and send a 502 (Bad

Gateway) response to the client. If this is a response message

received by a user agent, the user agent MUST close the

connection to the server and discard the received response.

5. If a valid Content-Length header field is present without

Transfer-Encoding, its decimal value defines the expected message

body length in octets. If the sender closes the connection or

the recipient times out before the indicated number of octets are

received, the recipient MUST consider the message to be

incomplete and close the connection.

6. If this is a request message and none of the above are true, then

the message body length is zero (no message body is present).

7. Otherwise, this is a response message without a declared message

body length, so the message body length is determined by the

number of octets received prior to the server closing the

connection.

参考链接

Transfer-Encoding 的作用

前端万字精华「浏览器简史及其核心原理详解」

今天来聊一聊程序员尤其是前端，离不开的工具「浏览器」。浏览器只要是学习过计算机的小伙伴都不陌生吧？它的主要功能就是向服务器发出请求，在浏览器窗口中展示HTML文档、PDF、图片、视频等网络内容。这些网络资源的位置由用户使用 URI（统一资源标示符）来指定。

继续阅读前端万字精华「浏览器简史及其核心原理详解」

Gradle 都做了哪些缓存？

前言

Gradle是Android的构建工具，它的主要目标就是实现快速的编译构建，而这主要就是通过缓存实现的。本文主要介绍Gradle的缓存机制，具体包括以下内容

Gradle缓存机制
Gradle内存缓存
Gradle项目缓存
Gradle本机缓存
Gradle远程缓存

继续阅读Gradle 都做了哪些缓存？

MMKV 高性能的数据存取框架解读

目标

了解MMKV

MMKV的基本应用

MMKV的原理概念

多进程设计思想

性能对比

源码解读

简介

MMKV 是基于 mmap 内存映射的 key-value 组件，底层序列化/反序列化使用 protobuf 实现，性能高，稳定性强。

官方文档：https://github.com/Tencent/MMKV/blob/master/README_CN.md

项目地址：https://github.com/Tencent/MMKV

继续阅读MMKV 高性能的数据存取框架解读

Flutter this and base files have different roots 问题

Flutter项目是能运行的，打开Flutter里面的Android项目才会报下面错误。

8:34    Gradle sync started
8:35    Gradle sync failed: Could not create task ':image_picker:generateDebugUnitTestConfig'.
            this and base files have different roots: D:\Pensoon\flutter_property_check_gd\build\image_picker and C:\Users\XXX\AppData\Roaming\Pub\Cache\hosted\pub.flutter-io.cn\image_picker-0.8.3+2\android. (52 s 230 ms)
8:35    Gradle sync started
8:35    Gradle sync failed: Could not create task ':image_picker:generateDebugUnitTestConfig'.
            this and base files have different roots: D:\Pensoon\flutter_property_check_gd\build\image_picker and C:\Users\XXX\AppData\Roaming\Pub\Cache\hosted\pub.flutter-io.cn\image_picker-0.8.3+2\android. (2 s 588 ms)

8:34 Gradle sync started

8:35 Gradle sync failed: Could not create task ':image_picker:generateDebugUnitTestConfig'.

this and base files have different roots: D:\Pensoon\flutter_property_check_gd\build\image_picker and C:\Users\XXX\AppData\Roaming\Pub\Cache\hosted\pub.flutter-io.cn\image_picker-0.8.3+2\android. (52 s 230 ms)

8:35 Gradle sync started

8:35 Gradle sync failed: Could not create task ':image_picker:generateDebugUnitTestConfig'.

$ .\gradlew clean build
Configuration on demand is an incubating feature.

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':url_launcher_android:test'.
> Could not create task ':url_launcher_android:testDebugUnitTest'.
   > this and base files have different roots: D:\Source\xxxx\build\url_launcher_android and C:\Users\Administrator\AppData\Local\Pub\Cache\hosted\pub.flutter-io.cn\url_launcher_android-6.0.25\android.

$ .\gradlew clean build

Configuration on demand is an incubating feature.

FAILURE: Build failed with an exception.

* What went wrong:

Could not determine the dependencies of task ':url_launcher_android:test'.

> Could not create task ':url_launcher_android:testDebugUnitTest'.

> this and base files have different roots: D:\Source\xxxx\build\url_launcher_android and C:\Users\Administrator\AppData\Local\Pub\Cache\hosted\pub.flutter-io.cn\url_launcher_android-6.0.25\android.

报错的项目配置信息如下：

buildscript {
    ext.kotlin_version = '1.8.0'
    repositories {
        maven { url "https://maven.aliyun.com/nexus/content/groups/public/" }
        maven { url "https://maven.aliyun.com/nexus/content/repositories/google" }
        google()
        mavenCentral()
    }

    dependencies {
        classpath 'com.android.tools.build:gradle:7.4.2'
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
    }
}

allprojects {
    repositories {
        maven { url "https://maven.aliyun.com/nexus/content/groups/public" }
        maven { url "https://maven.aliyun.com/nexus/content/repositories/google" }
        google()
        mavenCentral()
    }
}

rootProject.buildDir = '../build'
subprojects {
    project.buildDir = "${rootProject.buildDir}/${project.name}"
}
subprojects {
    project.evaluationDependsOn(':app')
}

task clean(type: Delete) {
    delete rootProject.buildDir
}

buildscript {

ext.kotlin_version = '1.8.0'

repositories {

maven { url "https://maven.aliyun.com/nexus/content/groups/public/" }

maven { url "https://maven.aliyun.com/nexus/content/repositories/google" }

google()

mavenCentral()

}

dependencies {

classpath 'com.android.tools.build:gradle:7.4.2'

classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"

}

allprojects {

repositories {

maven { url "https://maven.aliyun.com/nexus/content/groups/public" }

maven { url "https://maven.aliyun.com/nexus/content/repositories/google" }

google()

mavenCentral()

}

rootProject.buildDir = '../build'

subprojects {

project.buildDir = "${rootProject.buildDir}/${project.name}"

}

subprojects {

project.evaluationDependsOn(':app')

}

task clean(type: Delete) {

delete rootProject.buildDir

}

Flutter一开始Android build是没问题的，开发着突然就报这个下面的错误，开始怀疑是不是有什么缓存啥的，然后各种排除都没找到什么原因，后面想着降版本吧，kotlin降了没用，后面尝试最后一个Gradle降版本竟然成功了。

build.gradle文件

// 报错版本
classpath 'com.android.tools.build:gradle:7.0.0'

// 报错版本 
classpath 'com.android.tools.build:gradle:7.3.1'

// 报错版本
classpath 'com.android.tools.build:gradle:7.4.2'

// 解决版本
classpath 'com.android.tools.build:gradle:4.1.3'

// 解决版本 
classpath 'com.android.tools.build:gradle:4.2.2'

// 报错版本

classpath 'com.android.tools.build:gradle:7.0.0'

// 报错版本

classpath 'com.android.tools.build:gradle:7.3.1'

// 报错版本

classpath 'com.android.tools.build:gradle:7.4.2'

// 解决版本

classpath 'com.android.tools.build:gradle:4.1.3'

// 解决版本

classpath 'com.android.tools.build:gradle:4.2.2'

2023 年 3 月
一	二	三	四	五	六	日
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31