Compare commits

..

69 Commits

Author SHA1 Message Date
scito
157e247416 enable python 3.12 2023-10-06 12:06:02 +02:00
scito
a0bf28f4cc enable python 3.12 2023-10-06 12:06:02 +02:00
scito
fd8b165212 enable python 3.12 2023-10-06 12:06:02 +02:00
scito
a467955b13 bump versions 2023-10-05 20:19:31 +02:00
scito
eea0179b5e bump versions 2023-09-22 11:56:20 +02:00
scito
d20d709d6e bump versions 2023-09-05 13:10:47 +02:00
scito
a0c21273e8 Bump mypy and protobuf 2023-08-19 08:39:22 +02:00
dependabot[bot]
c7ab0b8ef4 Bump nuitka from 1.7.9 to 1.7.10
Bumps [nuitka](https://github.com/Nuitka/Nuitka) from 1.7.9 to 1.7.10.
- [Changelog](https://github.com/Nuitka/Nuitka/blob/develop/Changelog.rst)
- [Commits](https://github.com/Nuitka/Nuitka/commits)

---
updated-dependencies:
- dependency-name: nuitka
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-18 12:42:22 +02:00
scito
8e70ae9da9 bump protobuf to 24.0, cv2 and mypy 2023-08-12 12:44:45 +02:00
dependabot[bot]
511bb4a91c Bump types-protobuf from 4.23.0.2 to 4.23.0.3
Bumps [types-protobuf](https://github.com/python/typeshed) from 4.23.0.2 to 4.23.0.3.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-protobuf
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-12 10:02:46 +02:00
scito
15cbd77b07 improve numpy typing 2023-08-06 21:45:56 +02:00
scito
2610afe5d8 remove Python 3.7 workarounds, fixes #103 2023-08-06 19:29:59 +02:00
scito
8a4e2e3641 upgrade docker debian from bullseye (11) to bookworm (12) 2023-08-06 19:29:59 +02:00
scito
a0cf4246c1 remove deprecated pkg_resources package 2023-08-06 19:29:59 +02:00
dependabot[bot]
b3c0912e1f Bump nuitka from 1.7.5 to 1.7.9
Bumps [nuitka](https://github.com/Nuitka/Nuitka) from 1.7.5 to 1.7.9.
- [Changelog](https://github.com/Nuitka/Nuitka/blob/develop/Changelog.rst)
- [Commits](https://github.com/Nuitka/Nuitka/commits)

---
updated-dependencies:
- dependency-name: nuitka
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-04 17:20:06 +02:00
dependabot[bot]
1d8b49efcc Bump pylint from 2.17.4 to 2.17.5
Bumps [pylint](https://github.com/pylint-dev/pylint) from 2.17.4 to 2.17.5.
- [Release notes](https://github.com/pylint-dev/pylint/releases)
- [Commits](https://github.com/pylint-dev/pylint/compare/v2.17.4...v2.17.5)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-04 15:28:28 +02:00
dependabot[bot]
f846ee40af Bump flake8 from 6.0.0 to 6.1.0
Bumps [flake8](https://github.com/pycqa/flake8) from 6.0.0 to 6.1.0.
- [Commits](https://github.com/pycqa/flake8/compare/6.0.0...6.1.0)

---
updated-dependencies:
- dependency-name: flake8
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-04 09:11:18 +02:00
dependabot[bot]
f8b0283bdd Bump mypy-protobuf from 3.4.0 to 3.5.0
Bumps [mypy-protobuf](https://github.com/nipunn1313/mypy-protobuf) from 3.4.0 to 3.5.0.
- [Changelog](https://github.com/nipunn1313/mypy-protobuf/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nipunn1313/mypy-protobuf/compare/v3.4.0...v3.5.0)

---
updated-dependencies:
- dependency-name: mypy-protobuf
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-03 19:31:41 +02:00
dependabot[bot]
bcbf189289 Bump setuptools-git-versioning from 1.13.3 to 1.13.4
Bumps [setuptools-git-versioning](https://github.com/dolfinus/setuptools-git-versioning) from 1.13.3 to 1.13.4.
- [Release notes](https://github.com/dolfinus/setuptools-git-versioning/releases)
- [Changelog](https://github.com/dolfinus/setuptools-git-versioning/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/dolfinus/setuptools-git-versioning/compare/v1.13.3...v1.13.4)

---
updated-dependencies:
- dependency-name: setuptools-git-versioning
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-03 11:34:40 +02:00
scito
7c295e0963 fix #106: avoid UnboundLocalError for count_Xotp_entries: check has_Xotp 2023-07-23 14:32:42 +02:00
dependabot[bot]
6b72fad3c7 Bump wheel from 0.40.0 to 0.41.0
Bumps [wheel](https://github.com/pypa/wheel) from 0.40.0 to 0.41.0.
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](https://github.com/pypa/wheel/compare/0.40.0...0.41.0)

---
updated-dependencies:
- dependency-name: wheel
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-23 14:07:00 +02:00
dependabot[bot]
c937bede32 Bump types-protobuf from 4.23.0.1 to 4.23.0.2
Bumps [types-protobuf](https://github.com/python/typeshed) from 4.23.0.1 to 4.23.0.2.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-protobuf
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-23 14:06:49 +02:00
dependabot[bot]
7402263dfe Bump nuitka from 1.7.3 to 1.7.5
Bumps [nuitka](https://github.com/Nuitka/Nuitka) from 1.7.3 to 1.7.5.
- [Changelog](https://github.com/Nuitka/Nuitka/blob/develop/Changelog.rst)
- [Commits](https://github.com/Nuitka/Nuitka/compare/1.7.3...1.7.5)

---
updated-dependencies:
- dependency-name: nuitka
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-23 13:21:05 +02:00
dependabot[bot]
b721571101 Bump nuitka from 1.7.1 to 1.7.3
Bumps [nuitka](https://github.com/Nuitka/Nuitka) from 1.7.1 to 1.7.3.
- [Changelog](https://github.com/Nuitka/Nuitka/blob/develop/Changelog.rst)
- [Commits](https://github.com/Nuitka/Nuitka/compare/1.7.1...1.7.3)

---
updated-dependencies:
- dependency-name: nuitka
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-16 18:23:42 +02:00
scito
72bac4d951 bump versions: cv2 4.8 compatibility 2023-07-07 12:43:38 +02:00
scito
6c16d28194 bump versions 2023-06-22 20:48:42 +02:00
dependabot[bot]
87d2d4a05d Bump protobuf from 4.23.1 to 4.23.2
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 4.23.1 to 4.23.2.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v4.23.1...v4.23.2)

---
updated-dependencies:
- dependency-name: protobuf
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-02 08:55:11 +02:00
scito
ad3f184a56 upgrade pyinstaller 3.11 2023-05-21 08:21:11 +02:00
scito
994f96f749 Upgrade protobuf 23.0 2023-05-14 19:37:03 +02:00
scito
8177a97d39 bump pylint version 2023-05-07 09:05:36 +02:00
scito
0c0ac817b1 Bump versions 2023-05-05 10:31:56 +02:00
dependabot[bot]
5133711179 Bump nuitka from 1.5.6 to 1.5.7
Bumps [nuitka](https://github.com/Nuitka/Nuitka) from 1.5.6 to 1.5.7.
- [Release notes](https://github.com/Nuitka/Nuitka/releases)
- [Changelog](https://github.com/Nuitka/Nuitka/blob/develop/Changelog.rst)
- [Commits](https://github.com/Nuitka/Nuitka/compare/1.5.6...1.5.7)

---
updated-dependencies:
- dependency-name: nuitka
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-03 22:04:44 +02:00
dependabot[bot]
5c02e7d478 Bump pylint from 2.17.2 to 2.17.3
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.17.2 to 2.17.3.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.17.2...v2.17.3)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-03 20:08:24 +02:00
scito
6932793236 ci: disable Python 3.10 testing only for macos 2023-04-28 13:33:10 +02:00
scito
9c3edea75c fix/workaround build on macos: exclude 3.10 on macos in ci
See issue https://github.com/actions/setup-python/issues/649
2023-04-15 10:43:26 +02:00
scito
0ec1314b3d add nuitka to local build.sh; bump versions 2023-04-15 10:43:26 +02:00
scito
414e80c469 bump versions 2023-03-25 14:51:58 +01:00
scito
7f6959783f bump versions 2023-03-10 08:48:50 +01:00
scito
419f65fdea improve pyzbar missing warning; bump libs 2023-03-05 11:53:52 +01:00
dependabot[bot]
9783a5f4f2 Bump setuptools-git-versioning from 1.13.1 to 1.13.2
Bumps [setuptools-git-versioning](https://github.com/dolfinus/setuptools-git-versioning) from 1.13.1 to 1.13.2.
- [Release notes](https://github.com/dolfinus/setuptools-git-versioning/releases)
- [Changelog](https://github.com/dolfinus/setuptools-git-versioning/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/dolfinus/setuptools-git-versioning/compare/v1.13.1...v1.13.2)

---
updated-dependencies:
- dependency-name: setuptools-git-versioning
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-27 17:39:21 +01:00
scito
eccb1948c7 improve README for macOS 2023-02-26 21:19:45 +01:00
scito
01a38e570c optional build of exe and pipenv; update README 2023-02-25 09:24:37 +01:00
Damon McMinn
91b9490afb docs: update README with correct Fedora zbar dep
`zbar` is the available dependency: https://packages.fedoraproject.org/pkgs/zbar/zbar/
2023-02-22 20:06:06 +01:00
scito
11e530e04a docker no-install-recommends 2023-02-18 17:03:18 +01:00
scito
9e334748ac move Dockerfiles to docker/; update README 2023-02-18 17:03:18 +01:00
dependabot[bot]
05f3d89c42 Bump mypy from 1.0.0 to 1.0.1
Bumps [mypy](https://github.com/python/mypy) from 1.0.0 to 1.0.1.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v1.0.0...v1.0.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-17 21:54:48 +01:00
scito
6bf02d045c write txt file in cv2 and cmd; upgrade protobuf 4.22.0 2023-02-17 19:22:58 +01:00
scito
5555dda9c2 release: update release notes text 2023-02-12 19:51:19 +01:00
scito
eb8ea3330f release: improve macos docs after macos tests
- build macos dmg
    - however, do not upload macos dmg to assets as it cannot be
      executed since the app and installer are not signed and notarized
- upload artifacts
- tested executable on macOS
2023-02-12 19:00:29 +01:00
scito
75e5d2671f fix: rename vscode workspace 2023-02-12 19:00:29 +01:00
scito
5e94c43140 ci_release: fix get tag name (checkout repo); print meta values 2023-02-12 19:00:29 +01:00
kvascev
2ef34a2d69 Fix typo in README.md
Change "form" to "from"
2023-02-08 17:09:05 +01:00
scito
8df61f6ed7 ci: no docker login for dependabot 2023-02-07 10:51:07 +01:00
dependabot[bot]
def5d58af0 Bump mypy from 0.991 to 1.0.0
Bumps [mypy](https://github.com/python/mypy) from 0.991 to 1.0.0.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.991...v1.0.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-06 21:16:05 +01:00
scito
ca23497c9a docu clean ups 2023-02-06 20:42:54 +01:00
scito
ebfe0dc58d ci_release: add release notes table 2023-02-06 11:54:09 +01:00
scito
78118c73e8 make zbar lib optional
- refactor: global variable renaming
2023-02-05 21:24:57 +01:00
scito
4fc5559e15 add version number to asset executables 2023-02-05 21:24:57 +01:00
scito
7faf530863 reproducible builds, build.sh: linux/arm64 2023-02-05 21:24:57 +01:00
scito
709aa111be ci_release: build linux arm64 exe 2023-02-03 22:14:53 +01:00
scito
fe60acb24f release: upload hashes as binary to keep eol 2023-01-30 20:44:33 +01:00
scito
5c36f07f41 ci_release: build exe without upload on PR 2023-01-30 14:39:50 +01:00
scito
861d7d0da8 build buster image in ci_docker for speeding up releases 2023-01-30 11:28:07 +01:00
scito
63fc21cc90 ci_release: upload hashes and run on pull_request and run scheduled 2023-01-30 11:28:07 +01:00
scito
d234cf4112 cv2: fix save as csv/json/keepass 2023-01-30 11:28:07 +01:00
scito
970dbd3759 cv2: save as csv/json/keppass by key command 2023-01-29 16:50:55 +01:00
scito
197347a3e9 update README 2023-01-27 20:58:33 +01:00
scito
365d5ac432 use hub.docker for linux exe build
since it is more stable and thus faster
2023-01-26 01:14:45 +01:00
scito
88ff584e47 ci: workaround for failing builds 2023-01-26 00:18:41 +01:00
24 changed files with 1641 additions and 969 deletions

View File

@@ -20,7 +20,7 @@ jobs:
strategy:
matrix:
python-version: ["3.x", "3.11", "3.10", "3.9", "3.8", "3.7"]
python-version: ["3.x", "3.12", "3.11", "3.10", "3.9", "3.8"]
platform: [ubuntu-latest, macos-latest, windows-latest]
# exclude:
@@ -60,8 +60,7 @@ jobs:
if: matrix.python-version == '3.x' && matrix.platform == 'ubuntu-latest'
- name: Test with pytest
run: pytest
if: (matrix.python-version != '3.x' || matrix.platform != 'ubuntu-latest')
# && matrix.platform != 'macos-latest'
if: (matrix.python-version != '3.x' || matrix.platform != 'ubuntu-latest') && (matrix.python-version != '3.10' && matrix.platform != 'macos-latest')
- name: Test with pytest (with code coverage)
run: pytest --cov=extract_otp_secrets_test --junitxml=pytest.xml --cov-report=term-missing | tee pytest-coverage.txt
if: matrix.python-version == '3.x' && matrix.platform == 'ubuntu-latest'
@@ -72,6 +71,6 @@ jobs:
pytest-coverage-path: ./pytest-coverage.txt
junitxml-path: ./pytest.xml
if: |
matrix.python-version == '3.x' && matrix.platform == 'ubuntu-latest'
false && matrix.python-version == '3.x' && matrix.platform == 'ubuntu-latest'
&& !contains(github.ref, 'refs/tags/')

View File

@@ -25,7 +25,7 @@ on:
jobs:
build-and-push-docker-debian-image:
name: Build Docker Bullseye image and push to repositories
name: Build Docker Bookworm image and push to repositories
# run only when code is compiling and tests are passing
runs-on: ubuntu-latest
@@ -50,15 +50,22 @@ jobs:
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2
# Workaround for failing builds: https://github.com/docker/build-push-action/issues/761#issuecomment-1383822381
# TODO remove workaround when fixed
with:
driver-opts: |
image=moby/buildkit:v0.10.6
- name: Login to DockerHub
uses: docker/login-action@v2
if: github.secret_source == 'Actions'
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to Github Packages
uses: docker/login-action@v2
if: github.secret_source == 'Actions'
with:
registry: ghcr.io
username: ${{ github.actor }}
@@ -72,22 +79,31 @@ jobs:
# relative path to the place where source code with Dockerfile is located
# TODO file:, move to docker/
context: .
file: Dockerfile
file: docker/Dockerfile
# builder: ${{ steps.buildx.outputs.name }}
# Note: tags has to be all lower-case
build-args: |
BASE_IMAGE=python:3.12-slim-bookworm
pull: true
tags: |
scit0/extract_otp_secrets:latest
scit0/extract_otp_secrets:bullseye
scit0/extract_otp_secrets:bookworm
ghcr.io/scito/extract_otp_secrets:latest
ghcr.io/scito/extract_otp_secrets:bullseye
ghcr.io/scito/extract_otp_secrets:bookworm
# build on feature branches, push only on master branch
push: ${{ github.ref == 'refs/heads/master' }}
push: ${{ github.ref == 'refs/heads/master' && github.secret_source == 'Actions'}}
- name: Image digest
# TODO upload digests to assets
run: |
echo "extract_otp_secrets: ${{ steps.docker_build_qr_reader_latest.outputs.digest }}"
echo "extract_otp_secrets digests: ${{ steps.docker_build_qr_reader_latest.outputs.digest }}"
echo "${{ steps.docker_build_qr_reader_latest.outputs.digest }}" > digests.txt
- name: Save docker digests as artifacts
if: github.ref == 'refs/heads/master'
uses: actions/upload-artifact@v3
with:
name: debian_digests
path: digests.txt
build-and-push-docker-alpine-image:
name: Build Docker Alpine image and push to repositories
@@ -118,12 +134,14 @@ jobs:
- name: Login to DockerHub
uses: docker/login-action@v2
if: github.secret_source == 'Actions'
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to Github Packages
uses: docker/login-action@v2
if: github.secret_source == 'Actions'
with:
registry: ghcr.io
username: ${{ github.actor }}
@@ -136,7 +154,7 @@ jobs:
# relative path to the place where source code with Dockerfile is located
platforms: linux/amd64,linux/arm64
context: .
file: Dockerfile_only_txt
file: docker/Dockerfile_only_txt
# builder: ${{ steps.buildx.outputs.name }}
# Note: tags has to be all lower-case
pull: true
@@ -146,12 +164,96 @@ jobs:
ghcr.io/scito/extract_otp_secrets:only-txt
ghcr.io/scito/extract_otp_secrets:alpine
# build on feature branches, push only on master branch
push: ${{ github.ref == 'refs/heads/master' }}
push: ${{ github.ref == 'refs/heads/master' && github.secret_source == 'Actions'}}
build-args: |
RUN_TESTS=true
- name: Image digest
# TODO upload digests to assets
run: |
echo "extract_otp_secrets:only-txt: ${{ steps.docker_build_only_txt.outputs.digest }}"
echo "extract_otp_secrets:only-txt digests: ${{ steps.docker_build_only_txt.outputs.digest }}"
echo "${{ steps.docker_build_qr_reader_latest.outputs.digest }}" > digests.txt
- name: Save docker digests as artifacts
if: github.ref == 'refs/heads/master'
uses: actions/upload-artifact@v3
with:
name: alpine_digests
path: digests.txt
build-and-push-docker-bullseye-image:
name: Build Docker Bullseye image (for PyInstsaller) and push to repositories
# run only when code is compiling and tests are passing
runs-on: ubuntu-latest
# steps to perform in job
steps:
- name: Checkout code
uses: actions/checkout@v3
# avoid building if there are testing errors
- name: Run smoke test
run: |
sudo apt-get install -y libzbar0
python -m pip install --upgrade pip
pip install -U -r requirements-dev.txt
pip install -U .
pytest
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
# setup Docker build action
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2
# Workaround for failing builds: https://github.com/docker/build-push-action/issues/761#issuecomment-1383822381
# TODO remove workaround when fixed
with:
driver-opts: |
image=moby/buildkit:v0.10.6
- name: Login to DockerHub
uses: docker/login-action@v2
if: github.secret_source == 'Actions'
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to Github Packages
uses: docker/login-action@v2
if: github.secret_source == 'Actions'
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GHCR_IO_TOKEN }}
- name: "Build image from Bullseye and push to GitHub Container Registry"
id: docker_build_bullseye
if: github.ref == 'refs/heads/master'
uses: docker/build-push-action@v3
with:
platforms: linux/amd64,linux/arm64
# relative path to the place where source code with Dockerfile is located
context: .
file: docker/Dockerfile
# builder: ${{ steps.buildx.outputs.name }}
build-args: |
BASE_IMAGE=python:3.12-slim-bullseye
# Note: tags has to be all lower-case
pull: true
tags: |
scit0/extract_otp_secrets:bullseye
push: ${{ github.secret_source == 'Actions' }}
- name: Image digest
# TODO upload digests to assets
run: |
echo "extract_otp_secrets digests: ${{ steps.docker_build_qr_reader_latest.outputs.digest }}"
echo "${{ steps.docker_build_qr_reader_latest.outputs.digest }}" > digests.txt
- name: Save docker digests as artifacts
if: github.ref == 'refs/heads/master'
uses: actions/upload-artifact@v3
with:
name: bullseye_digests
path: digests.txt

View File

@@ -1,10 +1,8 @@
name: release
# https://data-dive.com/multi-os-deployment-in-cloud-using-pyinstaller-and-github-actions
# https://github.com/actions/create-release (archived)
# https://github.com/actions/upload-artifact
# https://github.com/actions/download-artifact
# https://github.com/actions/upload-release-asset (archived)
# https://github.com/docker/metadata-action
# https://github.com/marketplace/actions/generate-release-hashes
@@ -22,9 +20,14 @@ name: release
# https://peps.python.org/pep-0440/
# https://semver.org/
# macOS:
# https://pyinstaller.org/en/stable/usage.html#building-macos-app-bundles
# https://github.com/pyinstaller/pyinstaller/wiki/Recipe-OSX-Code-Signing
# Build matrix:
# - Linux x86_64 glibc 2.35: ubuntu-latest
# - Linux x86_64 glibc 2.34: extract_otp_secrets:buster
# - Linux x86_64 glibc 2.31: extract_otp_secrets:bullseye
# - Linux x86_64 glibc 2.36: extract_otp_secrets:bookworm
# - Windows x86_64: windows-latest
# - MacOS x86_64: macos-11
# - Linux x86_64 glibc 2.28: extract_otp_secrets:buster
@@ -36,27 +39,47 @@ on:
push:
tags:
- 'v*' # Push events to matching v*, i.e. v1.0, v20.15.10
pull_request:
schedule:
# Run weekly on default branch
- cron: '47 4 * * 6'
jobs:
create-release:
name: Create Release
runs-on: ubuntu-latest
outputs:
date: ${{ steps.meta.outputs.date }}
version: ${{ steps.meta.outputs.version }}
inline_version: ${{ steps.meta.outputs.inline_version }}
tag_name: ${{ steps.meta.outputs.tag_name }}
tag_message: ${{ steps.meta.outputs.tag_message }}
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set meta data
id: meta
# Writing to env with >> $GITHUB_ENV is an alternative
run: |
echo "date=$(TZ=Europe/Zurich date +'%d.%m.%Y')" >> $GITHUB_OUTPUT
echo "version=${TAG_NAME/v/}" >> $GITHUB_OUTPUT
echo "inline_version=${TAG_NAME/v/_}" >> $GITHUB_OUTPUT
echo "tag_name=${{ github.ref_name }}" >> $GITHUB_OUTPUT
echo "tag_message=$(git tag -l --format='%(contents:subject)' ${{ github.ref_name }})" >> $GITHUB_OUTPUT
env:
TAG_NAME: ${{ github.ref_name }}
PYTHONHASHSEED: 31
- name: Create Release
id: create_release
if: startsWith(github.ref, 'refs/tags/v')
# https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28#create-a-release
run: |
# https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28#create-a-release
echo "date: ${{ steps.meta.outputs.date }}"
echo "version: ${{ steps.meta.outputs.version }}"
echo "inline_version: ${{ steps.meta.outputs.inline_version }}"
echo "tag_name: ${{ steps.meta.outputs.tag_name }}"
echo "tag_message: ${{ steps.meta.outputs.tag_message }}"
response=$(curl \
-X POST \
-H "Accept: application/vnd.github+json" \
@@ -65,26 +88,37 @@ jobs:
https://api.github.com/repos/scito/extract_otp_secrets/releases \
--silent \
--show-error \
-d '{"tag_name":"${{ github.ref }}","target_commitish":"master","name":"${{ steps.meta.outputs.version }} - ${{ steps.meta.outputs.date }}","body":"${{ steps.meta.outputs.tag_message }}","draft":true,"prerelease":false,"generate_release_notes":true}')
-d '{"tag_name":"${{ github.ref }}","target_commitish":"master","name":"${{ steps.meta.outputs.version }} - ${{ steps.meta.outputs.date }}","body":"${{ steps.meta.outputs.tag_message }}\n\n## Executables\n\nDownload the executable for your platform and execute it, see [README.md](https://github.com/scito/extract_otp_secrets#readme)\n\n | Executable | Description |\n | --- | --- |\n | extract_otp_secrets${{ steps.meta.outputs.inline_version }}_linux_x86_64 | Linux x86_64/amd64 (glibc >= 2.31) |\n | extract_otp_secrets${{ steps.meta.outputs.inline_version }}_linux_arm64 | Linux arm64 (glibc >= 2.31) |\n | extract_otp_secrets${{ steps.meta.outputs.inline_version }}_win_x86_64.exe | Windows x86_64/amd64/x64 |\n | extract_otp_secrets${{ steps.meta.outputs.inline_version }}_win_arm64.exe | N/A |\n | extract_otp_secrets${{ steps.meta.outputs.inline_version }}_macos_x86_64.dmg | N/A, see [README.md](https://github.com/scito/extract_otp_secrets#readme) |\n | extract_otp_secrets${{ steps.meta.outputs.inline_version }}_macos_x86_64.pkg | N/A, see [README.md](https://github.com/scito/extract_otp_secrets#readme) |\n | extract_otp_secrets${{ steps.meta.outputs.inline_version }}_macos_x86_64 | MacOS x86_64/amd64 (bare executable, see [README.md](https://github.com/scito/extract_otp_secrets#readme); optional libzbar must be installed manually, see [README.md](https://github.com/scito/extract_otp_secrets#readme)) |\n | extract_otp_secrets${{ steps.meta.outputs.inline_version }}_macos_arm64 | N/A |\n","draft":true,"prerelease":false,"generate_release_notes":true}')
echo upload_url=$(jq '.upload_url' <<< "$response") >> $GITHUB_OUTPUT
echo $(jq -r '.upload_url' <<< "$response") > release_url.txt
echo $(jq -r '.id' <<< "$response") > release_id.txt
- name: Save Release URL File for publish
if: startsWith(github.ref, 'refs/tags/v')
uses: actions/upload-artifact@v3
with:
name: release_url
path: release_url.txt
- name: Save asset upload id for publish
if: startsWith(github.ref, 'refs/tags/v')
uses: actions/upload-artifact@v3
with:
name: release_id
path: release_id.txt
build-and-push-docker-image:
name: Build Linux release in docker container
build-linux-executable-in-docker:
name: Build ${{ matrix.PLATFORM }} release in docker container
# run only when code is compiling and tests are passing
runs-on: ubuntu-latest
needs: create-release
strategy:
matrix:
include:
- PLATFORM: linux/amd64
EXE: extract_otp_secrets_linux_x86_64
ASSET_NAME: extract_otp_secrets${{ needs.create-release.outputs.inline_version }}_linux_x86_64
- PLATFORM: linux/arm64
EXE: extract_otp_secrets_linux_arm64
ASSET_NAME: extract_otp_secrets${{ needs.create-release.outputs.inline_version }}_linux_arm64
# steps to perform in job
steps:
@@ -107,74 +141,72 @@ jobs:
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2
# Workaround for failing builds: https://github.com/docker/build-push-action/issues/761#issuecomment-1383822381
# TODO remove workaround when fixed
with:
driver-opts: |
image=moby/buildkit:v0.10.6
- name: Login to DockerHub
uses: docker/login-action@v2
if: github.secret_source == 'Actions'
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to Github Packages
uses: docker/login-action@v2
if: github.secret_source == 'Actions'
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GHCR_IO_TOKEN }}
- name: "Build image from Buster and push to GitHub Container Registry"
id: docker_build_buster
uses: docker/build-push-action@v3
with:
platforms: linux/amd64,linux/arm64
# relative path to the place where source code with Dockerfile is located
# TODO file:, move to docker/
context: .
file: Dockerfile
# builder: ${{ steps.buildx.outputs.name }}
build-args: |
BASE_IMAGE=python:3.11-slim-buster
# Note: tags has to be all lower-case
pull: true
tags: |
scit0/extract_otp_secrets:buster
push: true
# # https://stackoverflow.com/a/61155718/1663871
# - name: Build docker images
# run: docker build -t local < .devcontainer/Dockerfile
# - name: Run tests
# run: docker run -it -v $PWD:/srv -w/srv local make test
- name: Image digest
# TODO upload digests to assets
run: |
echo "extract_otp_secrets: ${{ steps.docker_build_buster.outputs.digest }}"
echo "extract_otp_secrets: ${{ steps.docker_build_bullseye.outputs.digest }}"
- name: Run Pyinstaller in container
# TODO use local docker image https://stackoverflow.com/a/61155718/1663871
# https://github.com/multiarch/qemu-user-static
# https://hub.docker.com/r/multiarch/qemu-user-static/
- name: Run Pyinstaller in container for ${{ matrix.EXE }}
run: |
# TODO use local docker image https://stackoverflow.com/a/61155718/1663871
docker run --pull always --entrypoint /bin/bash --rm -v "$(pwd)":/files -w /files scit0/extract_otp_secrets:buster -c 'apt-get update && apt-get -y install binutils && pip install -U -r /files/requirements.txt && pip install pyinstaller && pyinstaller -y --add-data /usr/local/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile --name extract_otp_secrets_linux_x86_64 --distpath /files/dist/ /files/src/extract_otp_secrets.py'
docker run --pull always --rm --privileged multiarch/qemu-user-static --reset -p yes
docker run --platform ${{ matrix.PLATFORM }} --pull always --entrypoint /bin/bash --rm -v "$(pwd)":/files -w /files scit0/extract_otp_secrets:bullseye -c 'apt-get update && apt-get -y install binutils && pip install -U -r /files/requirements.txt && pip install pyinstaller && PYTHONHASHSEED=31 && pyinstaller -y --add-data /usr/local/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile --name ${{ matrix.EXE }} --distpath /files/dist/ /files/src/extract_otp_secrets.py'
- name: Smoke tests
- name: Smoke tests linux/amd64
if: matrix.PLATFORM == 'linux/amd64'
run: |
dist/extract_otp_secrets_linux_x86_64 -V
dist/extract_otp_secrets_linux_x86_64 -h
dist/extract_otp_secrets_linux_x86_64 example_export.png
dist/extract_otp_secrets_linux_x86_64 - < example_export.txt
dist/extract_otp_secrets_linux_x86_64 --qr ZBAR example_export.png
dist/extract_otp_secrets_linux_x86_64 --qr QREADER example_export.png
dist/extract_otp_secrets_linux_x86_64 --qr QREADER_DEEP example_export.png
dist/extract_otp_secrets_linux_x86_64 --qr CV2 example_export.png
dist/extract_otp_secrets_linux_x86_64 --qr CV2_WECHAT example_export.png
dist/${{ matrix.EXE }} -V
dist/${{ matrix.EXE }} -h
dist/${{ matrix.EXE }} --debug
dist/${{ matrix.EXE }} example_export.png
dist/${{ matrix.EXE }} - < example_export.txt
dist/${{ matrix.EXE }} --qr ZBAR example_export.png
dist/${{ matrix.EXE }} --qr QREADER example_export.png
dist/${{ matrix.EXE }} --qr QREADER_DEEP example_export.png
dist/${{ matrix.EXE }} --qr CV2 example_export.png
dist/${{ matrix.EXE }} --qr CV2_WECHAT example_export.png
- name: Smoke tests linux/arm64
if: matrix.PLATFORM == 'linux/arm64'
run: |
docker run --platform ${{ matrix.PLATFORM }} --pull always --entrypoint /bin/bash --rm -v "$(pwd)":/files -w /files scit0/extract_otp_secrets -c 'dist/${{ matrix.EXE }} -V && dist/${{ matrix.EXE }} -h && dist/${{ matrix.EXE }} example_export.png && dist/${{ matrix.EXE }} - < example_export.txt && dist/${{ matrix.EXE }} --qr ZBAR example_export.png && dist/${{ matrix.EXE }} --qr QREADER example_export.png && dist/${{ matrix.EXE }} --qr QREADER_DEEP example_export.png && dist/${{ matrix.EXE }} --qr CV2 example_export.png && dist/${{ matrix.EXE }} --qr CV2_WECHAT example_export.png'
- name: Load Release URL File from release job
if: startsWith(github.ref, 'refs/tags/v')
uses: actions/download-artifact@v3
with:
name: release_url
- name: Display structure of files
run: ls -R
- name: Upload EXE to artifacts
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.EXE }}
path: dist/${{ matrix.EXE }}
- name: Upload Release Asset
id: upload-release-asset
# TODO only for tags
if: startsWith(github.ref, 'refs/tags/v')
run: |
response=$(curl \
-X POST \
@@ -184,11 +216,11 @@ jobs:
-H "X-GitHub-Api-Version: 2022-11-28" \
--silent \
--show-error \
--data-binary @dist/extract_otp_secrets_linux_x86_64 \
$(cat release_url.txt)=extract_otp_secrets_linux_x86_64)
--data-binary @dist/${{ matrix.EXE }} \
$(cat release_url.txt)=${{ matrix.ASSET_NAME }})
build:
name: Build packages
build-native-executables:
name: Build native packages
needs: create-release
runs-on: ${{ matrix.os }}
strategy:
@@ -201,34 +233,33 @@ jobs:
# TODO add --manifest
# TODO find more elegant solution for pyzbar\libiconv.dll and pyzbar\libzbar-64.dll
# Files of Visual C++ 2013 Redistributable Package: https://support.microsoft.com/en-us/topic/update-for-visual-c-2013-redistributable-package-d8ccd6a5-4e26-c290-517b-8da6cfdf4f10
CMD_BUILD: |
pyinstaller -y --add-data "$($Env:pythonLocation)\__yolo_v3_qr_detector;__yolo_v3_qr_detector" --add-binary "$($Env:pythonLocation)\Lib\site-packages\pyzbar\libiconv.dll;pyzbar" --add-binary "$($Env:pythonLocation)\Lib\site-packages\pyzbar\libzbar-64.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\msvcr120.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\msvcp120.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\vcamp120.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\vcomp120.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\vccorlib120.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120u.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120chs.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120cht.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120deu.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120enu.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120esn.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120fra.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120ita.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120jpn.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120kor.dll;pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120rus.dll;pyzbar" --onefile --version-file build\file_version_info.txt src\extract_otp_secrets.py
OUT_FILE_NAME: extract_otp_secrets.exe
ASSET_NAME: extract_otp_secrets_win_x86_64.exe
EXE: extract_otp_secrets.exe
ASSET_NAME: extract_otp_secrets${{ needs.create-release.outputs.inline_version }}_win_x86_64.exe
ASSET_MIME: application/vnd.microsoft.portable-executable
UPLOAD: true
CMD_BUILD: |
pyinstaller -y --add-data "$($Env:pythonLocation)\__yolo_v3_qr_detector:__yolo_v3_qr_detector" --add-binary "$($Env:pythonLocation)\Lib\site-packages\pyzbar\libiconv.dll:pyzbar" --add-binary "$($Env:pythonLocation)\Lib\site-packages\pyzbar\libzbar-64.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\msvcr120.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\msvcp120.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\vcamp120.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\vcomp120.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\vccorlib120.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120u.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120chs.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120cht.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120deu.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120enu.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120esn.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120fra.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120ita.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120jpn.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120kor.dll:pyzbar" --add-binary "$($Env:WinDir)\system32\mfc120rus.dll:pyzbar" --onefile --version-file build\win_file_version_info.txt --name extract_otp_secrets.exe src\extract_otp_secrets.py
- os: macos-11
TARGET: macos
# TODO add --icon
# TODO add --osx-bundle-identifier
# TODO add --codesign-identity
# TODO add --osx-entitlements-file
# TODO https://pyinstaller.org/en/stable/spec-files.html#spec-file-options-for-a-macos-bundle
# TODO --target-arch universal2
CMD_BUILD: |
pyinstaller -y --add-data $macos_python_path/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile --argv-emulation src/extract_otp_secrets.py
OUT_FILE_NAME: extract_otp_secrets
ASSET_NAME: extract_otp_secrets_macos_x86_64
ASSET_MIME: application/x-newton-compatible-pkg
# https://pyinstaller.org/en/stable/spec-files.html#spec-file-options-for-a-macos-bundle
EXE: extract_otp_secrets
ASSET_NAME: extract_otp_secrets${{ needs.create-release.outputs.inline_version }}_macos_x86_64
DMG: extract_otp_secrets.dmg
ASSET_NAME_DMG: extract_otp_secrets${{ needs.create-release.outputs.inline_version }}_macos_x86_64.dmg
ASSET_MIME: application/octet-stream
UPLOAD: true
CMD_BUILD: |
VERSION_STR=$(setuptools-git-versioning) COPYRIGHT_YEARS='2020-2023' envsubst < installer/extract_otp_secrets_macos_template.spec > extract_otp_secrets_macos.spec
pyinstaller -y extract_otp_secrets_macos.spec
installer/build_dmg.sh
- os: ubuntu-latest
TARGET: linux
CMD_BUILD: |
pyinstaller -y --add-data $pythonLocation/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile src/extract_otp_secrets.py
OUT_FILE_NAME: extract_otp_secrets
ASSET_NAME: extract_otp_secrets_linux_x86_64_ubuntu_latest
EXE: extract_otp_secrets_ubuntu
ASSET_NAME: extract_otp_secrets${{ needs.create-release.outputs.inline_version }}_linux_x86_64_ubuntu_latest
ASSET_MIME: application/x-executable
UPLOAD: false
CMD_BUILD: |
pyinstaller -y --add-data $pythonLocation/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile --name extract_otp_secrets_ubuntu src/extract_otp_secrets.py
steps:
- name: Output path
if: runner.os == 'Windows'
@@ -239,11 +270,11 @@ jobs:
- uses: actions/checkout@v3
- name: Set macos macos_python_path
# TODO use variable for Python version
run: echo "macos_python_path=/Library/Frameworks/Python.framework/Versions/3.11" >> $GITHUB_ENV
- name: Set up Python 3.11
run: echo "macos_python_path=/Library/Frameworks/Python.framework/Versions/3.12" >> $GITHUB_ENV
- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: 3.11
python-version: 3.12
check-latest: true
- name: Install zbar shared lib for QReader (Linux)
if: runner.os == 'Linux'
@@ -252,39 +283,45 @@ jobs:
- name: Install zbar shared lib for QReader (macOS)
if: runner.os == 'macOS'
run: |
brew install zbar
brew install zbar create-dmg
- name: Install dependencies
# TODO fix --use-pep517
run: |
python -m pip install --upgrade pip
pip install -U -r requirements-dev.txt
pip install -U .
- name: Create Windows file_version_info.txt
- name: Create Windows win_file_version_info.txt
shell: bash
run: |
mkdir -p build/
VERSION_STR=$(setuptools-git-versioning) VERSION_MAJOR=$(cut -d '.' -f 1 <<< "$(setuptools-git-versioning)") VERSION_MINOR=$(cut -d '.' -f 2 <<< "$(setuptools-git-versioning)") VERSION_PATCH=$(echo $(cut -d '.' -f 3 <<< "$(setuptools-git-versioning)") | sed -E -n "s/^([0-9]+).*/\1/p") VERSION_BUILD=$(echo $(cut -d '.' -f 3 <<< "$(setuptools-git-versioning)") | sed -E -n -e"s/^[0-9]+.+/99/p")$(($(git rev-list --count $(git tag | sort -V -r | sed '1!d')..HEAD))) YEARS='2020-2023' envsubst < file_version_info_template.txt > build/file_version_info.txt
VERSION_STR=$(setuptools-git-versioning) VERSION_MAJOR=$(cut -d '.' -f 1 <<< "$(setuptools-git-versioning)") VERSION_MINOR=$(cut -d '.' -f 2 <<< "$(setuptools-git-versioning)") VERSION_PATCH=$(echo $(cut -d '.' -f 3 <<< "$(setuptools-git-versioning)") | sed -E -n "s/^([0-9]+).*/\1/p") VERSION_BUILD=$(echo $(cut -d '.' -f 3 <<< "$(setuptools-git-versioning)") | sed -E -n -e"s/^[0-9]+.+/99/p")$(($(git rev-list --count $(git tag | sort -V -r | sed '1!d')..HEAD))) COPYRIGHT_YEARS='2020-2023' envsubst < installer/win_file_version_info_template.txt > build/win_file_version_info.txt
- name: Build with pyinstaller for ${{ matrix.TARGET }}
env:
# Reproducible build: https://pyinstaller.org/en/stable/advanced-topics.html#creating-a-reproducible-build
PYTHONHASHSEED: 31
run: ${{ matrix.CMD_BUILD }}
- name: Smoke tests for generated exe (general)
run: |
dist/${{ matrix.OUT_FILE_NAME }} -V
dist/${{ matrix.OUT_FILE_NAME }} -h
dist/${{ matrix.OUT_FILE_NAME }} example_export.png
dist/${{ matrix.OUT_FILE_NAME }} --qr ZBAR example_export.png
dist/${{ matrix.OUT_FILE_NAME }} --qr QREADER example_export.png
dist/${{ matrix.OUT_FILE_NAME }} --qr QREADER_DEEP example_export.png
dist/${{ matrix.OUT_FILE_NAME }} --qr CV2 example_export.png
dist/${{ matrix.OUT_FILE_NAME }} --qr CV2_WECHAT example_export.png
dist/${{ matrix.EXE }} -V
dist/${{ matrix.EXE }} -h
dist/${{ matrix.EXE }} --debug
dist/${{ matrix.EXE }} example_export.png
dist/${{ matrix.EXE }} --qr ZBAR example_export.png
dist/${{ matrix.EXE }} --qr QREADER example_export.png
dist/${{ matrix.EXE }} --qr QREADER_DEEP example_export.png
dist/${{ matrix.EXE }} --qr CV2 example_export.png
dist/${{ matrix.EXE }} --qr CV2_WECHAT example_export.png
- name: Smoke tests for generated exe (stdin)
if: runner.os != 'Windows'
run: |
dist/${{ matrix.OUT_FILE_NAME }} - < example_export.txt
dist/${{ matrix.EXE }} - < example_export.txt
- name: Load Release URL File from release job
if: startsWith(github.ref, 'refs/tags/v')
uses: actions/download-artifact@v3
with:
name: release_url
- name: Load Release Id File from release job
if: startsWith(github.ref, 'refs/tags/v')
uses: actions/download-artifact@v3
with:
name: release_id
@@ -292,14 +329,82 @@ jobs:
run: ls -R
- name: Set meta data
id: meta
if: startsWith(github.ref, 'refs/tags/v')
shell: bash
run: |
cat release_url.txt
echo "release_url=$(cat release_url.txt)" >> $GITHUB_OUTPUT
echo "release_id=$(cat release_id.txt)" >> $GITHUB_OUTPUT
echo "upload_url=https://uploads.github.com/repos/scito/extract_otp_secrets/releases/$(cat release_id.txt)/assets?name=" >> $GITHUB_OUTPUT
- name: Upload EXE to artifacts
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.EXE }}
path: dist/${{ matrix.EXE }}
- name: Upload DMG to artifacts
uses: actions/upload-artifact@v3
if: runner.os == 'macOS'
with:
name: ${{ matrix.DMG }}
path: dist/${{ matrix.DMG }}
- name: Upload Release Asset
id: upload-release-asset
if: ${{ matrix.UPLOAD }}
if: matrix.UPLOAD && startsWith(github.ref, 'refs/tags/v')
run: |
curl -X POST -H "Accept: application/vnd.github+json" -H "Content-Type: ${{ matrix.ASSET_MIME }}" -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "X-GitHub-Api-Version: 2022-11-28" --show-error --data-binary @dist/${{ matrix.OUT_FILE_NAME }} ${{ steps.meta.outputs.upload_url }}=${{ matrix.ASSET_NAME }}
curl -X POST -H "Accept: application/vnd.github+json" -H "Content-Type: ${{ matrix.ASSET_MIME }}" -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "X-GitHub-Api-Version: 2022-11-28" --show-error --data-binary @dist/${{ matrix.EXE }} ${{ steps.meta.outputs.upload_url }}=${{ matrix.ASSET_NAME }}
- name: Upload Release Asset DMG (macOS)
id: upload-release-asset-dmg
if: false && matrix.UPLOAD && startsWith(github.ref, 'refs/tags/v') && runner.os == 'macOS'
run: |
curl -X POST -H "Accept: application/vnd.github+json" -H "Content-Type: ${{ matrix.ASSET_MIME }}" -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "X-GitHub-Api-Version: 2022-11-28" --show-error --data-binary @dist/${{ matrix.DMG }} ${{ steps.meta.outputs.upload_url }}=${{ matrix.ASSET_NAME_DMG }}
upload-hashes:
name: Upload hashes
if: startsWith(github.ref, 'refs/tags/v')
needs:
- build-linux-executable-in-docker
- build-native-executables
runs-on: ubuntu-latest
steps:
- name: Load Release Id File from release job
uses: actions/download-artifact@v3
with:
name: release_id
- name: Set meta data
id: meta
run: |
echo "release_id=$(cat release_id.txt)" >> $GITHUB_OUTPUT
echo "upload_url=https://uploads.github.com/repos/scito/extract_otp_secrets/releases/$(cat release_id.txt)/assets?name=" >> $GITHUB_OUTPUT
- name: Calculate and upload hashes from assets
run: |
GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}
for asset_url in $(curl \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer $GITHUB_TOKEN"\
-H "X-GitHub-Api-Version: 2022-11-28" \
--silent \
--show-error \
https://api.github.com/repos/scito/extract_otp_secrets/releases/${{ steps.meta.outputs.release_id }}/assets |
jq -r '.[].url'); do
echo "Download $asset_url"
name=$(curl \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer $GITHUB_TOKEN"\
-H "X-GitHub-Api-Version: 2022-11-28" \
--output-dir assets \
-L \
$asset_url |
jq -r '.name')
curl \
-H "Accept: application/octet-stream" \
-H "Authorization: Bearer $GITHUB_TOKEN"\
-H "X-GitHub-Api-Version: 2022-11-28" \
--create-dirs \
--output-dir assets \
-L \
-o $name \
$asset_url
done
(cd assets/ && sha256sum * > ../sha256_hashes.txt)
curl -X POST -H "Accept: application/vnd.github+json" -H "Content-Type: text/plain" -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "X-GitHub-Api-Version: 2022-11-28" --show-error --data-binary @sha256_hashes.txt ${{ steps.meta.outputs.upload_url }}=sha256_hashes.txt
(cd assets/ && sha512sum * > ../sha512_hashes.txt)
curl -X POST -H "Accept: application/vnd.github+json" -H "Content-Type: text/plain" -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "X-GitHub-Api-Version: 2022-11-28" --show-error --data-binary @sha512_hashes.txt ${{ steps.meta.outputs.upload_url }}=sha512_hashes.txt

12
.gitignore vendored
View File

@@ -20,8 +20,18 @@ dist/
pytest-coverage.txt
tests/reports/
dist_*/
*.spec
file_version_info_python.txt
file_version_info_explorer.txt
file_version_info.txt
assets/*
extract_otp_secrets_linux_arm64.spec
extract_otp_secrets_linux_x86_64_bullseye.spec
extract_otp_secrets_linux_x86_64_bookworm.spec
extract_otp_secrets_linux_x86_64.spec
extract_otp_secrets.spec
test.txt
extract_otp_secrets.build/
extract_otp_secrets.dist/
extract_otp_secrets.onefile-build/
extract_otp_secrets.bin

View File

@@ -7,8 +7,8 @@ name = "pypi"
colorama = ">=0.4.6"
opencv-contrib-python = "*"
# for macOS: opencv-contrib-python = "<=4.7.0"
# for PYTHON <= 3.7: typing_extensions = "*"
pillow = "*"
pyzbar = "*"
protobuf = "*"
qrcode = "*"
qreader = "<2.0.0"
@@ -19,6 +19,7 @@ flake8 = "*"
gfm-toc = "*"
mypy = "*"
mypy-protobuf = "*"
nuitka = "*"
pylint = "*"
pytest = "*"
pytest-cov = "*"

824
Pipfile.lock generated

File diff suppressed because it is too large Load Diff

109
README.md
View File

@@ -5,7 +5,7 @@
![coverage](https://img.shields.io/badge/coverage-94%25-brightgreen)
[![License](https://img.shields.io/github/license/scito/extract_otp_secrets)](https://github.com/scito/extract_otp_secrets/blob/master/LICENSE)
[![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/scito/extract_otp_secrets?sort=semver)](https://github.com/scito/extract_otp_secrets/releases/latest)
![python versions](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9%20%7C%203.10%20%7C%203.11-blue)
![python versions](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)
[![Docker image](https://img.shields.io/badge/docker-image-blue)](https://hub.docker.com/repository/docker/scit0/extract_otp_secrets/general)
[![Linux](https://img.shields.io/badge/os-linux-yellow)](https://github.com/scito/extract_otp_secrets/releases/latest)
[![Windows](https://img.shields.io/badge/os-windows-yellow)](https://github.com/scito/extract_otp_secrets/releases/latest)
@@ -14,7 +14,7 @@
[![Stand With Ukraine](https://raw.githubusercontent.com/vshymanskyy/StandWithUkraine/main/badges/StandWithUkraine.svg)](https://stand-with-ukraine.pp.ua)
<!-- ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/protobuf)
[![GitHub Pipenv locked Python version](https://img.shields.io/github/pipenv/locked/python-version/scito/extract_otp_secrets)](https://github.com/scito/extract_otp_secrets/blob/master/Pipfile.lock)
![protobuf version](https://img.shields.io/badge/protobuf-4.21.12-informational)-->
![protobuf version](https://img.shields.io/badge/protobuf-4.24.4-informational)-->
<!-- [![Github all releases](https://img.shields.io/github/downloads/scito/extract_otp_secrets/total.svg)](https://GitHub.com/scito/extract_otp_secrets/releases/) -->
@@ -36,16 +36,17 @@ The secrets can be exported to JSON or CSV, or printed as QR codes to console or
## Table of contents
- [Download binary executable (🆕 since v2.1)](#download-binary-executable--since-v21)
- [Download and run binary executable (🆕 since v2.1)](#download-and-run-binary-executable--since-v21)
- [MacOS](#macos)
- [Usage](#usage)
- [Capture QR codes from camera (🆕 since version 2.0)](#capture-qr-codes-from-camera--since-version-20)
- [With builtin QR decoder from image files (🆕 since version 2.0)](#with-builtin-qr-decoder-from-image-files--since-version-20)
- [With external QR decoder app from text files](#with-external-qr-decoder-app-from-text-files)
- [Installation of Python script (recommend for developers or advanced users)](#installation-of-python-script-recommend-for-developers-or-advanced-users)
- [Installation of shared system libraries](#installation-of-shared-system-libraries)
- [Installation of Python script (recommended for developers or advanced users)](#installation-of-python-script-recommended-for-developers-or-advanced-users)
- [Installation of optional shared system libraries (recommended)](#installation-of-optional-shared-system-libraries-recommended)
- [Program help: arguments and options](#program-help-arguments-and-options)
- [Examples](#examples)
- [Printing otp secrets form text file](#printing-otp-secrets-form-text-file)
- [Printing otp secrets from text file](#printing-otp-secrets-form-text-file)
- [Printing otp secrets from image file](#printing-otp-secrets-from-image-file)
- [Writing otp secrets to csv file](#writing-otp-secrets-to-csv-file)
- [Writing otp secrets to json file](#writing-otp-secrets-to-json-file)
@@ -86,22 +87,46 @@ The secrets can be exported to JSON or CSV, or printed as QR codes to console or
- [Related projects](#related-projects)
</details>
## Download binary executable (🆕 since v2.1)
## Download and run binary executable (🆕 since v2.1)
1. Download executable for your platform from [latest release](https://github.com/scito/extract_otp_secrets/releases/latest), see assets
2. Start executable by clicking or from command line
2. Linux and macOS: Set executable bit for the downloaded file, e.g in terminal with `chmod +x extract_otp_secrets_X.Y.Z_OS_ARCH`
3. Start executable by clicking or from command line (macOS: startable only from command line, see [below](#macos))
:heavy_check_mark: Everything is just packed in one executable.
:heavy_check_mark: No installation needed, neither Python nor any dependencies have to be installed.
:heavy_check_mark: Easy and convenient
:information_source: There is a delay after starting the executable since the files have internally to be unpacked.
:information_source: There is a delay after starting the executable since the files have internally to be unpacked.
:information_source: If you are a developer, you might prefer to run the Python script directly, see [Installation](#installation-of-python-script-recommended-for-developers-or-advanced-users)
> :warning: Some antivirus tools may show a virus or trojan alert for the executable.
> This alert is a false positive.
> This is a known problem for executables generated by PyInstaller.
> If you have any doubt, please use directly the [Python script](#installation-of-python-script-recommend-for-developers-or-advanced-users).
> If you have any doubt, please use directly the [Python script](#installation-of-python-script-recommended-for-developers-or-advanced-users).
> :information_source: The executables are not signed. Thus, the operating system may show a warning about download from unknown source.
:information_source: The executables are not signed. Thus, the operating system may show a warning about download from unknown source.
### MacOS
> Beginning in macOS 10.15, all software built after June 1, 2019, and distributed with Developer ID must be notarized. However, you arent required to notarize software that you distribute through the Mac App Store because the App Store submission process already includes equivalent security checks. <small>[developer.apple.com](https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution)</small>
:x: Unfortunately, I cannot provide a signed and notarized installable application for macOS as .dmg or .pkg. Apple is not Open Source friendly and requires a yearly Developer ID subscription. I am not willing to pay [USD 99 per year](https://developer.apple.com/support/compare-memberships/) to Apple for this little open source tool.
However, the bare executable can be executed from the command line:
1. Download executable for macOS platform from [latest release](https://github.com/scito/extract_otp_secrets/releases/latest), see assets
2. Open `Terminal` application
3. Change to Downloads folder in Terminal: `cd $HOME/Downloads`
4. Set executable bit for the downloaded file: `chmod +x extract_otp_secrets_X.Y.Z_macos_x86_64`
5. Start executable from command line: `./extract_otp_secrets_X.Y.Z_macos_x86_64`
:information_source: Replace `X.Y.Z` in above commands with the version number of your downloaded file, e.g. `extract_otp_secrets_2.4.0_macos_x86_64`
:information_source: If Rosetta2 emulation is installed, these steps work also for M1 and M2 Apple Silicon processors and the program can be executed directly.
> :warning: It seems the GUI mode is not working in Terminal on macOS. In tests no [GUI window](#usage) was opened. (Remarks and hints about macOS are welcome since I do not know macOS.)
## Usage
@@ -123,6 +148,14 @@ Detected QR codes are surrounded with a frame. The color of the frame indicates
* Red: The QR code is detected and decoded, but could not be successfully extracted. This is the case if a QR code not containing OTP data is captured.
* Magenta: The QR code is detected, but could not be decoded. The QR code should be presented better to the camera or another QR reader could be used.
Key commands:
* Space: change QR code reader
* C: save as csv file (🆕 since v2.2)
* J: save as json file (🆕 since v2.2)
* K: save as KeePass csv file (🆕 since v2.2)
* ESC, ENTER, Q: quit the program
The secrets are printed by default to the console. [Set program parameters](#program-help-arguments-and-options) for other types of output, e.g. `--csv exported_secrets.csv`.
### With builtin QR decoder from image files (🆕 since version 2.0)
@@ -150,7 +183,7 @@ extract_otp_secrets example_export.txt
```
7. Remove unencrypted files with secrets from your computer and mobile.
## Installation of Python script (recommend for developers or advanced users)
## Installation of Python script (recommended for developers or advanced users)
```bash
git clone https://github.com/scito/extract_otp_secrets.git
@@ -162,7 +195,7 @@ python src/extract_otp_secrets.py example_export.txt
In case this script is not starting properly, the debug mode can be activated by adding parameter `-d` in the command line.
### Installation of shared system libraries
### Installation of optional shared system libraries (recommended)
For reading QR codes with `ZBAR` QR reader, the zbar library must be installed.
If you do not use the `ZBAR` QR reader, you do not need to install the zbar shared library. Note: The `ZBAR` QR reader is the showed for me the best results and is thus default QR Reader.
@@ -179,7 +212,7 @@ For a detailed installation documentation of [pyzbar](https://github.com/Natural
#### Linux (Fedora)
sudo dnf install libzbar0
sudo dnf install zbar
#### Linux (Arch Linux)
@@ -201,7 +234,7 @@ OpenCV requires [Visual C++ redistributable 2015](https://www.microsoft.com/en-u
## Program help: arguments and options
<pre>usage: extract_otp_secrets.py [-h] [--csv FILE] [--keepass FILE] [--json FILE] [--printqr] [--saveqr DIR] [--camera NUMBER] [--qr {ZBAR,QREADER,QREADER_DEEP,CV2,CV2_WECHAT}] [-i] [--no-color] [--version] [-d | -v | -q] [infile ...]
<pre>usage: extract_otp_secrets.py [-h] [--csv FILE] [--keepass FILE] [--json FILE] [--txt FILE] [--printqr] [--saveqr DIR] [--camera NUMBER] [--qr {ZBAR,QREADER,QREADER_DEEP,CV2,CV2_WECHAT}] [-i] [--no-color] [--version] [-d | -v | -q] [infile ...]
Extracts one time password (OTP) secrets from QR codes exported by two-factor authentication (2FA) apps
If no infiles are provided, a GUI window starts and QR codes are captured from the camera.
@@ -215,8 +248,9 @@ options:
--csv FILE, -c FILE export csv file or - for stdout
--keepass FILE, -k FILE export totp/hotp csv file(s) for KeePass, - for stdout
--json FILE, -j FILE export json file or - for stdout
--printqr, -p print QR code(s) as text to the terminal (requires qrcode module)
--saveqr DIR, -s DIR save QR code(s) as images to the given folder (requires qrcode module)
--txt FILE, -t FILE export txt file or - for stdout
--printqr, -p print QR code(s) as text to the terminal
--saveqr DIR, -s DIR save QR code(s) as images to directory
--camera NUMBER, -C NUMBER camera number of system (default camera: 0)
--qr {ZBAR,QREADER,QREADER_DEEP,CV2,CV2_WECHAT}, -Q {ZBAR,QREADER,QREADER_DEEP,CV2,CV2_WECHAT}
QR reader (default: ZBAR)
@@ -284,11 +318,15 @@ python extract_otp_secrets.py = < example_export.png</pre>
* Free and open source
* Supports Google Authenticator exports (and compatible apps like Aegis Authenticator)
* Captures the the QR codes directly from the camera using different QR code libraries (based on OpenCV) (🆕 since v2.0)
* ZBAR: [pyzbar](https://github.com/NaturalHistoryMuseum/pyzbar) - fast and reliable, good for images and video capture (default and recommended)
* QREADER: [QReader](https://github.com/Eric-Canas/QReader)
* QREADER_DEEP: [QReader](https://github.com/Eric-Canas/QReader) - very slow in GUI
* ZBAR: [pyzbar](https://github.com/NaturalHistoryMuseum/pyzbar) - fast and reliable, good for images and video capture (default and recommended) [if [libzbar](#installation-of-optional-shared-system-libraries-recommended) is installed]
* QREADER: [QReader](https://github.com/Eric-Canas/QReader) [if [libzbar](#installation-of-optional-shared-system-libraries-recommended) is installed]
* QREADER_DEEP: [QReader](https://github.com/Eric-Canas/QReader) - very slow in GUI [if [libzbar](#installation-of-optional-shared-system-libraries-recommended) is installed]
* CV2: [QRCodeDetector](https://docs.opencv.org/4.x/de/dc3/classcv_1_1QRCodeDetector.html)
* CV2_WECHAT: [WeChatQRCode](https://docs.opencv.org/4.x/dd/d63/group__wechat__qrcode.html)
* Program usable as pure GUI application without any command line switches (🆕 since v2.2)
* Save otp secrets as csv file (🆕 since v2.2)
* Save otp secrets as json file (🆕 since v2.2)
* Save otp secrets as KeePass csv file(s) (🆕 since v2.2)
* Supports [TOTP](https://www.ietf.org/rfc/rfc6238.txt) and [HOTP](https://www.ietf.org/rfc/rfc4226.txt) standards
* Generates QR codes
* Exports to various formats:
@@ -309,9 +347,12 @@ python extract_otp_secrets.py = < example_export.png</pre>
* Prints errors and warnings to stderr (🆕 since v2.0)
* Prints colored output (🆕 since v2.0)
* Startable as executable (script, Python, and all dependencies packed in one executable) (🆕 since v2.1)
* extract_otp_secrets_linux_x86_64 (requires glibc >= 2.28)
* extract_otp_secrets_linux_x86_64 (requires glibc >= 2.31)
* extract_otp_secrets_linux_arm64 (requires glibc >= 2.31)
* extract_otp_secrets_win_x86_64.exe
* extract_otp_secrets_macos_x86_64 (untested)
* extract_otp_secrets_macos_x86_64 (optional [libzbar](#installation-of-optional-shared-system-libraries-recommended) needs to be installed manually if needed)
* extract_otp_secrets_macos_x86_64.dmg N/A, see [why](#macos)
* extract_otp_secrets_macos_x86_64.pkg N/A, see [why](#macos)
* Prebuilt Docker images provided for amd64 and arm64 (🆕 since v2.0)
* Many ways to run the script:
* Native Python
@@ -326,7 +367,8 @@ python extract_otp_secrets.py = < example_export.png</pre>
* macOS
* Windows
* Uses UTF-8 on all platforms
* Supports Python >= 3.7
* Supports Python >= 3.8
* Installation of shared system libraries is optional (🆕 since v2.3)
* Provides a debug mode (-d) for analyzing import problems
* Written in modern Python using type hints and following best practices
* All these features are backed by tests ran nightly
@@ -586,7 +628,7 @@ pip install -U -r requirements.txt
Build and run the app within the container:
```bash
docker build . -t extract_otp_secrets --pull --build-arg RUN_TESTS=false
docker build . -t extract_otp_secrets --pull -f docker/Dockerfile --build-arg RUN_TESTS=false
```
Run tests in docker container:
@@ -598,7 +640,7 @@ docker run --entrypoint /extract/run_pytest.sh --rm -v "$(pwd)":/files:ro extrac
#### Alpine (only text file processing)
```bash
docker build . -t extract_otp_secrets:only_txt --pull -f Dockerfile_only_txt --build-arg RUN_TESTS=false
docker build . -t extract_otp_secrets:only_txt --pull -f docker/Dockerfile_only_txt --build-arg RUN_TESTS=false
```
Run tests in docker container:
@@ -643,11 +685,18 @@ Build extract_otp_secrets project
Options:
-i Interactive mode, all steps must be confirmed
-C Ignore version check of protobuf/protoc
-D Do not build docker
-G Do not start extract_otp_secrets.py in GUI mode
-e Build exe
-n Build nuitka exe
-L Do not build local (exes)
-d Build docker
-a Build arm
-X Do not build x86_64
-B Do not build base
-V Do not run pipenv
-g Start extract_otp_secrets.py in GUI mode
-c Clean everything
-r Generate result files
-h, --help Help
-h, --help Show help and quit
```
## Technical background
@@ -659,7 +708,7 @@ Command for regeneration of Python code from proto3 message definition file (onl
protoc --plugin=protoc-gen-mypy=path/to/protoc-gen-mypy --python_out=src/protobuf_generated_python --mypy_out=src/protobuf_generated_python src/google_auth.proto
The generated protobuf Python code was generated by protoc 21.12 (https://github.com/protocolbuffers/protobuf/releases/tag/v21.12).
The generated protobuf Python code was generated by protoc 24.4 (https://github.com/protocolbuffers/protobuf/releases/tag/v24.4).
For Python type hint generation the [mypy-protobuf](https://github.com/nipunn1313/mypy-protobuf) package is used.
@@ -703,7 +752,7 @@ FileNotFoundError: Could not find module 'libiconv.dll' (or one of its dependenc
## Related projects
* [ZBar](https://github.com/mchehab/zbar) is an open source software suite for reading bar codes from various sources, including webcams.
* [Aegis Authenticator](https://github.com/beemdevelopment/Aegis) is a free, secure and open source 2FA app for Android.
* [Aegis Authenticator](https://github.com/beemdevelopment/Aegis) is a free, secure and open source 2FA app for Android. This app can scan Google export QR codes and export the secrets, e.g. as JSON. However, a second device is required.
* [pyzbar](https://github.com/NaturalHistoryMuseum/pyzbar) is a good QR code reader Python module
* [OpenCV](https://docs.opencv.org/4.x/) (CV2) Open Source Computer Vision library with [opencv-python](https://github.com/opencv/opencv-python)
* [Python QReader](https://github.com/Eric-Canas/QReader) Python QR code readers

689
build.sh
View File

@@ -77,9 +77,17 @@ interactive=false
ignore_version_check=true
clean=false
clean_flag=""
build_docker=true
run_gui=true
build_base=true
build_arm=false
build_x86_64=true
build_docker=false
build_local=true
build_exe=false
build_nuitka_exe=false
run_pipenv=true
run_gui=false
generate_result_files=false
PYTHONHASHSEED=31
while test $# -gt 0; do
case $1 in
@@ -91,14 +99,21 @@ while test $# -gt 0; do
echo "Options:"
echo "-i Interactive mode, all steps must be confirmed"
echo "-C Ignore version check of protobuf/protoc"
echo "-D Do not build docker"
echo "-G Do not start extract_otp_secrets.py in GUI mode"
echo "-e Build exe"
echo "-n Build nuitka exe"
echo "-L Do not build local (exe)"
echo "-d Build docker"
echo "-a Build arm"
echo "-X Do not build x86_64"
echo "-B Do not build base"
echo "-V Do not run pipenv"
echo "-g Start extract_otp_secrets.py in GUI mode"
echo "-c Clean everything"
echo "-r Generate result files"
echo "-h, --help Help"
echo "-h, --help Show help and quit"
quit
;;
-a)
-i)
interactive=true
shift
;;
@@ -106,12 +121,41 @@ while test $# -gt 0; do
ignore_version_check=false
shift
;;
-D)
build_docker=false
-B)
build_base=false
shift
;;
-G)
run_gui=false
-L)
build_local=false
build_base=false
shift
;;
-a)
build_arm=true
shift
;;
-X)
build_x86_64=false
shift
;;
-e)
build_exe=true
shift
;;
-n)
build_nuitka_exe=true
shift
;;
-d)
build_docker=true
shift
;;
-V)
run_pipenv=false
shift
;;
-g)
run_gui=true
shift
;;
-r)
@@ -173,296 +217,435 @@ if $clean; then
cmd="sudo pipenv --rm || true"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
cmd="$PIP install --use-pep517 -U -r requirements-dev.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
echo -e "\n\nChecking Protoc version..."
VERSION=$(curl -sL https://github.com/protocolbuffers/protobuf/releases/latest | grep -E "<title>" | perl -pe's%.*Protocol Buffers v(\d+\.\d+(\.\d+)?).*%\1%')
BASEVERSION=4
echo
OLDVERSION=$(cat $BIN/$DEST/.VERSION.txt || echo "")
echo -e "\nProtoc remote version $VERSION\n"
echo -e "Protoc local version: $OLDVERSION\n"
if [ "$OLDVERSION" != "$VERSION" ] || ! $ignore_version_check; then
echo "Upgrade protoc from $OLDVERSION to $VERSION"
NAME="protoc-$VERSION"
ARCHIVE="$NAME.zip"
mkdir -p $DOWNLOADS
# https://github.com/protocolbuffers/protobuf/releases/download/v21.6/protoc-21.6-linux-x86_64.zip
cmd="wget --trust-server-names https://github.com/protocolbuffers/protobuf/releases/download/v$VERSION/protoc-$VERSION-linux-x86_64.zip -O $DOWNLOADS/$ARCHIVE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="echo -e '\nSize [Byte]'; stat --printf='%s\n' $DOWNLOADS/$ARCHIVE; echo -e '\nMD5'; md5sum $DOWNLOADS/$ARCHIVE; echo -e '\nSHA256'; sha256sum $DOWNLOADS/$ARCHIVE;"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="mkdir -p $BIN/$NAME; unzip $DOWNLOADS/$ARCHIVE -d $BIN/$NAME"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="echo $VERSION > $BIN/$NAME/.VERSION.txt; echo $VERSION > $BIN/$NAME/.VERSION_$VERSION.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="[ -d $BIN/$DEST.old ] && rm -rf $BIN/$DEST.old || echo 'No old dir to delete'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="[ -d $BIN/$DEST ] && mv -iT $BIN/$DEST $BIN/$DEST.old || echo 'No previous dir to keep'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="mv -iT $BIN/$NAME $BIN/$DEST"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="rm $DOWNLOADS/$ARCHIVE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$BIN/$DEST/bin/protoc --plugin=protoc-gen-mypy=$HOME/.local/bin/protoc-gen-mypy --python_out=src/protobuf_generated_python --mypy_out=src/protobuf_generated_python --proto_path=src google_auth.proto"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Update README.md
cmd="perl -i -pe 's%proto(buf|c)([- ])(\d\.)?$OLDVERSION%proto\$1\$2\${3}$VERSION%g' README.md && perl -i -pe 's%(protobuf/releases/tag/v)$OLDVERSION%\${1}$VERSION%g' README.md"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
else
echo -e "\nVersion has not changed. Quit"
fi
# Upgrade pip requirements
cmd="pip install -U pip"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
$PIP --version
cmd="$PIP install --use-pep517 -U -r requirements.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Lint
LINT_OUT_FILE="tests/reports/flake8_results.txt"
cmd="$FLAKE8 . --count --select=E9,F63,F7,F82 --show-source --statistics | tee $LINT_OUT_FILE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$FLAKE8 . --count --exit-zero --max-complexity=10 --max-line-length=200 --statistics --exclude=.git,__pycache__,docs/source/conf.py,old,build,dist,protobuf_generated_python | tee -a $LINT_OUT_FILE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Type checking
TYPE_CHECK_OUT_FILE="tests/reports/mypy_results.txt"
cmd="$MYPY --install-types --non-interactive src/*.py tests/*.py"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# change to src as python -m mypy adds the current dir Python sys.path
# execute in a subshell in order not to loose the exit code and not to change the dir in the currrent shell
cmd="$MYPY --strict src/*.py tests/*.py | tee $TYPE_CHECK_OUT_FILE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Generate results files
if $generate_result_files; then
cmd="for color in '' '-n'; do for level in '' '-v' '-vv' '-vvv'; do $PYTHON src/extract_otp_secrets.py example_export.txt \$color \$level > tests/data/print_verbose_output\$color\$level.txt; done; done"
cmd="mkdir -p dist"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
# pip -e install
if $build_local; then
cmd="$PIP install --use-pep517 -U -r requirements-dev.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$PIP install -U -e ."
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
echo -e "\n\nChecking Protoc version..."
VERSION=$(curl -sL https://github.com/protocolbuffers/protobuf/releases/latest | grep -E "<title>" | perl -pe's%.*Protocol Buffers v(\d+\.\d+(\.\d+)?).*%\1%')
BASEVERSION=4
echo
cmd="extract_otp_secrets example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
OLDVERSION=$(cat $BIN/$DEST/.VERSION.txt || echo "")
echo -e "\nProtoc remote version $VERSION\n"
echo -e "Protoc local version: $OLDVERSION\n"
cmd="extract_otp_secrets - < example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
if [ "$OLDVERSION" != "$VERSION" ] || ! $ignore_version_check; then
echo "Upgrade protoc from $OLDVERSION to $VERSION"
# Test (needs module)
NAME="protoc-$VERSION"
ARCHIVE="$NAME.zip"
cmd="$PYTHON src/extract_otp_secrets.py example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
mkdir -p $DOWNLOADS
# https://github.com/protocolbuffers/protobuf/releases/download/v21.6/protoc-21.6-linux-x86_64.zip
cmd="wget --trust-server-names https://github.com/protocolbuffers/protobuf/releases/download/v$VERSION/protoc-$VERSION-linux-x86_64.zip -O $DOWNLOADS/$ARCHIVE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$PYTHON src/extract_otp_secrets.py - < example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="echo -e '\nSize [Byte]'; stat --printf='%s\n' $DOWNLOADS/$ARCHIVE; echo -e '\nMD5'; md5sum $DOWNLOADS/$ARCHIVE; echo -e '\nSHA256'; sha256sum $DOWNLOADS/$ARCHIVE;"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
COVERAGE_OUT_FILE="tests/reports/pytest-coverage.txt"
cmd="pytest --cov=extract_otp_secrets_test --junitxml=tests/reports/pytest.xml --cov-report html:tests/reports/html --cov-report=term-missing tests/ | tee $COVERAGE_OUT_FILE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="mkdir -p $BIN/$NAME; unzip $DOWNLOADS/$ARCHIVE -d $BIN/$NAME"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Pipenv
cmd="echo $VERSION > $BIN/$NAME/.VERSION.txt; echo $VERSION > $BIN/$NAME/.VERSION_$VERSION.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$PIP install -U pipenv"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="[ -d $BIN/$DEST.old ] && rm -rf $BIN/$DEST.old || echo 'No old dir to delete'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
$PIPENV --version
cmd="[ -d $BIN/$DEST ] && mv -iT $BIN/$DEST $BIN/$DEST.old || echo 'No previous dir to keep'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$PIPENV update && $PIPENV --rm && $PIPENV install"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="mv -iT $BIN/$NAME $BIN/$DEST"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
$PIPENV run python --version
cmd="rm $DOWNLOADS/$ARCHIVE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$PIPENV run pytest --cov=extract_otp_secrets_test tests/"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$BIN/$DEST/bin/protoc --plugin=protoc-gen-mypy=$HOME/.local/bin/protoc-gen-mypy --python_out=src/protobuf_generated_python --mypy_out=src/protobuf_generated_python --proto_path=src google_auth.proto"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Build wheel
# Update README.md
cmd="$PIP wheel ."
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="perl -i -pe 's%proto(buf|c)([- ])(\d\.)?$OLDVERSION%proto\$1\$2\${3}$VERSION%g' README.md && perl -i -pe 's%(protobuf/releases/tag/v)$OLDVERSION%\${1}$VERSION%g' README.md"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
else
echo -e "\nVersion has not changed. Quit"
fi
# Build executable
# Upgrade pip requirements
cmd="LOCAL_GLIBC_VERSION=$(ldd --version | sed '1!d' | sed -E 's/.* ([[:digit:]]+\.[[:digit:]]+)$/\1/')"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
echo "local glibc: $LOCAL_GLIBC_VERSION"
cmd="pip install -U pip"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="pyinstaller -y --add-data $HOME/.local/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile $clean_flag src/extract_otp_secrets.py"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
$PIP --version
cmd="dist/extract_otp_secrets -h"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$PIP install --use-pep517 -U -r requirements.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Generate README.md TOC
if $build_base; then
# Lint
cmd="gfm-toc -s 2 -e 3 -t -o README.md > docs/README_TOC.md"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
LINT_OUT_FILE="tests/reports/flake8_results.txt"
cmd="$FLAKE8 . --count --select=E9,F63,F7,F82 --show-source --statistics | tee $LINT_OUT_FILE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Update Code Coverage in README.md
cmd="$FLAKE8 . --count --exit-zero --max-complexity=10 --max-line-length=200 --statistics --exclude=.git,__pycache__,docs/source/conf.py,old,build,dist,protobuf_generated_python | tee -a $LINT_OUT_FILE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# https://github.com/marketplace/actions/pytest-coverage-comment
# Coverage-95%25-yellowgreen
echo -e "Update code coverage in README.md"
TOTAL_COVERAGE=$(cat $COVERAGE_OUT_FILE | grep 'TOTAL' | perl -ne 'print "$&" if /\b(\d{1,3})%/') && perl -i -pe "s/coverage-(\d{1,3}%)25-/coverage-${TOTAL_COVERAGE}25-/" README.md
# Type checking
TYPE_CHECK_OUT_FILE="tests/reports/mypy_results.txt"
cmd="$MYPY --install-types --non-interactive src/*.py tests/*.py"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# change to src as python -m mypy adds the current dir Python sys.path
# execute in a subshell in order not to loose the exit code and not to change the dir in the currrent shell
cmd="$MYPY --strict src/*.py tests/*.py | tee $TYPE_CHECK_OUT_FILE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Generate results files
if $generate_result_files; then
cmd="for color in '' '-n'; do for level in '' '-v' '-vv' '-vvv'; do $PYTHON src/extract_otp_secrets.py example_export.txt \$color \$level > tests/data/print_verbose_output\$color\$level.txt; done; done"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
# pip -e install
cmd="$PIP install -U -e ."
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="extract_otp_secrets example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="extract_otp_secrets - < example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Test (needs module)
cmd="$PYTHON src/extract_otp_secrets.py example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$PYTHON src/extract_otp_secrets.py - < example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
COVERAGE_OUT_FILE="tests/reports/pytest-coverage.txt"
cmd="pytest --cov=extract_otp_secrets_test --junitxml=tests/reports/pytest.xml --cov-report html:tests/reports/html --cov-report=term-missing tests/ | tee $COVERAGE_OUT_FILE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Pipenv
if $run_pipenv; then
cmd="$PIP install -U pipenv"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
$PIPENV --version
cmd="$PIPENV --rm && $PIPENV update && $PIPENV install"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
$PIPENV run python --version
cmd="$PIPENV run pytest --cov=extract_otp_secrets_test tests/"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
# Build wheel
cmd="$PIP wheel ."
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
# Build executable
if $build_exe; then
cmd="LOCAL_GLIBC_VERSION=$(ldd --version | sed '1!d' | sed -E 's/.* ([[:digit:]]+\.[[:digit:]]+)$/\1/')"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
echo "local glibc: $LOCAL_GLIBC_VERSION"
cmd="pyinstaller -y --specpath installer --add-data $HOME/.local/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile $clean_flag src/extract_otp_secrets.py"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="dist/extract_otp_secrets -h"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="dist/extract_otp_secrets --qr ZBAR example_export.png"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
# Build compiled executable
if $build_nuitka_exe; then
cmd="LOCAL_GLIBC_VERSION=$(ldd --version | sed '1!d' | sed -E 's/.* ([[:digit:]]+\.[[:digit:]]+)$/\1/')"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
echo "local glibc: $LOCAL_GLIBC_VERSION"
cmd="$PIP install -U pyqt5"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="$PYTHON -m nuitka --enable-plugin=tk-inter --enable-plugin=pyqt5 --include-data-dir=$HOME/.local/__yolo_v3_qr_detector/=__yolo_v3_qr_detector/ --onefile --output-dir=build/nuitka --output-filename=extract_otp_secrets_compiled src/extract_otp_secrets.py"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="cp build/nuitka/extract_otp_secrets_compiled dist/"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="dist/extract_otp_secrets_compiled -h"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="dist/extract_otp_secrets_compiled --qr ZBAR example_export.png"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
# Generate README.md TOC
cmd="gfm-toc -s 2 -e 3 -t -o README.md > docs/README_TOC.md"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Update Code Coverage in README.md
# https://github.com/marketplace/actions/pytest-coverage-comment
# Coverage-95%25-yellowgreen
echo -e "Update code coverage in README.md"
TOTAL_COVERAGE=$(cat $COVERAGE_OUT_FILE | grep 'TOTAL' | perl -ne 'print "$&" if /\b(\d{1,3})%/') && perl -i -pe "s/coverage-(\d{1,3}%)25-/coverage-${TOTAL_COVERAGE}25-/" README.md
# create Windows win_file_version_info.txt
cmd="VERSION_STR=$(setuptools-git-versioning) VERSION_MAJOR=$(cut -d '.' -f 1 <<< "$(setuptools-git-versioning)") VERSION_MINOR=$(cut -d '.' -f 2 <<< "$(setuptools-git-versioning)") VERSION_PATCH=$(cut -d '.' -f 3 <<< "$(setuptools-git-versioning)") VERSION_BUILD=$(($(git rev-list --count $(git tag | sort -V -r | sed '1!d')..HEAD))) COPYRIGHT_YEARS='2020-2023' envsubst < installer/win_file_version_info_template.txt > build/win_file_version_info.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# create macOS extract_otp_secrets_macos.spec from extract_otp_secrets_macos_template.spec
cmd="VERSION_STR=$(setuptools-git-versioning) COPYRIGHT_YEARS='2020-2023' envsubst < installer/extract_otp_secrets_macos_template.spec > build/extract_otp_secrets_macos.spec"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
if $build_docker; then
# Build docker
# Build Dockerfile_only_txt (Alpine)
cmd="docker build . -t extract_otp_secrets_only_txt -t extract_otp_secrets:only-txt -t extract_otp_secrets:alpine -f Dockerfile_only_txt --pull --build-arg RUN_TESTS=false"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
if $build_x86_64; then
# Build Dockerfile_only_txt (Alpine)
cmd="docker build . -t extract_otp_secrets_only_txt -t extract_otp_secrets:only-txt -t extract_otp_secrets:alpine -f docker/Dockerfile_only_txt --pull --build-arg RUN_TESTS=false"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -v \"$(pwd)\":/files:ro extract_otp_secrets_only_txt example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -v \"$(pwd)\":/files:ro extract_otp_secrets_only_txt example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -i -v \"$(pwd)\":/files:ro extract_otp_secrets_only_txt - < example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -i -v \"$(pwd)\":/files:ro extract_otp_secrets_only_txt - < example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --entrypoint /extract/run_pytest.sh --rm -v \"$(pwd)\":/files:ro extract_otp_secrets_only_txt extract_otp_secrets_test.py -k 'not qreader' -vvv --relaxed"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --entrypoint /extract/run_pytest.sh --rm -v \"$(pwd)\":/files:ro extract_otp_secrets_only_txt extract_otp_secrets_test.py -k 'not qreader' -vvv --relaxed"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Build extract_otp_secrets (Debian Bullseye)
cmd="docker build . -t extract_otp_secrets -t extract_otp_secrets:bullseye --pull --build-arg RUN_TESTS=false"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Build extract_otp_secrets (Debian Bookworm)
cmd="docker build . -t extract_otp_secrets -t extract_otp_secrets:bookworm --pull -f docker/Dockerfile --build-arg RUN_TESTS=false"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -v \"$(pwd)\":/files:ro extract_otp_secrets example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -v \"$(pwd)\":/files:ro extract_otp_secrets example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="cat example_export.txt | docker run --rm -i -v \"$(pwd)\":/files:ro extract_otp_secrets - -c - > example_output.csv"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="cat example_export.txt | docker run --rm -i -v \"$(pwd)\":/files:ro extract_otp_secrets - -c - > example_output.csv"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -i -v \"$(pwd)\":/files:ro extract_otp_secrets = < example_export.png"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -i -v \"$(pwd)\":/files:ro extract_otp_secrets = < example_export.png"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --entrypoint /extract/run_pytest.sh --rm -v \"$(pwd)\":/files:ro extract_otp_secrets"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --entrypoint /extract/run_pytest.sh --rm -v \"$(pwd)\":/files:ro extract_otp_secrets"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Build extract_otp_secrets (Debian Buster)
cmd="docker build . -t extract_otp_secrets:buster --pull --build-arg RUN_TESTS=false --build-arg BASE_IMAGE=python:3.11-slim-buster"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Build extract_otp_secrets (Debian Bullseye)
cmd="docker build . -t extract_otp_secrets:bullseye -t extract_otp_secrets:bullseye-x86_64 --pull -f docker/Dockerfile --build-arg RUN_TESTS=false --build-arg BASE_IMAGE=python:3.12-slim-bullseye"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -v \"$(pwd)\":/files:ro extract_otp_secrets:buster example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -v \"$(pwd)\":/files:ro extract_otp_secrets:bullseye example_export.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="cat example_export.txt | docker run --rm -i -v \"$(pwd)\":/files:ro extract_otp_secrets:buster - -c - > example_output.csv"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="cat example_export.txt | docker run --rm -i -v \"$(pwd)\":/files:ro extract_otp_secrets:bullseye - -c - > example_output.csv"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -i -v \"$(pwd)\":/files:ro extract_otp_secrets:buster = < example_export.png"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --rm -i -v \"$(pwd)\":/files:ro extract_otp_secrets:bullseye = < example_export.png"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --entrypoint /extract/run_pytest.sh --rm -v \"$(pwd)\":/files:ro extract_otp_secrets:buster"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --entrypoint /extract/run_pytest.sh --rm -v \"$(pwd)\":/files:ro extract_otp_secrets:bullseye"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Build executable from Docker latest
# sed "1!d" is workaround for head -n 1 since it head procduces exit code != 0
BULLSEYE_GLIBC_VERSION=$(docker run --entrypoint /bin/bash --rm extract_otp_secrets -c 'ldd --version | sed "1!d" | sed -E "s/.* ([[:digit:]]+\.[[:digit:]]+)$/\1/"')
echo "Bullseye glibc: $BULLSEYE_GLIBC_VERSION"
# Build executable from Docker latest
# sed "1!d" is workaround for head -n 1 since it head procduces exit code != 0
BOOKWORM_GLIBC_VERSION=$(docker run --entrypoint /bin/bash --rm extract_otp_secrets -c 'ldd --version | sed "1!d" | sed -E "s/.* ([[:digit:]]+\.[[:digit:]]+)$/\1/"')
echo "Bookworm glibc: $BOOKWORM_GLIBC_VERSION"
fi
cmd="docker run --entrypoint /bin/bash --rm -v \"$(pwd)\":/files -w /files extract_otp_secrets -c 'apt-get update && apt-get -y install binutils && pip install -U -r /files/requirements.txt && pip install pyinstaller && pyinstaller -y --add-data /usr/local/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile --name extract_otp_secrets_linux_x86_64_bullseye --distpath /files/dist/ /files/src/extract_otp_secrets.py'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
if $build_arm; then
# build linux/arm64
cmd="docker run --pull always --rm --privileged multiarch/qemu-user-static --reset -p yes"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="dist/extract_otp_secrets_linux_x86_64_bullseye -h"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Build extract_otp_secrets (Debian Bullseye)
cmd="docker buildx build --platform=linux/arm64 . -t extract_otp_secrets:bullseye-arm64 --pull -f docker/Dockerfile --build-arg RUN_TESTS=false --build-arg BASE_IMAGE=python:3.12-slim-bullseye"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
# Build executable from Docker buster
BUSTER_GLIBC_VERSION=$(docker run --entrypoint /bin/bash --rm extract_otp_secrets:buster -c 'ldd --version | sed "1!d" | sed -E "s/.* ([[:digit:]]+\.[[:digit:]]+)$/\1/"')
echo "Bullseye glibc: $BUSTER_GLIBC_VERSION"
if $build_exe; then
if $build_x86_64; then
cmd="docker run --platform linux/amd64 --entrypoint /bin/bash --rm -v \"$(pwd)\":/files -w /files extract_otp_secrets -c 'apt-get update && apt-get -y install binutils && pip install -U pip && pip install -U -r /files/requirements.txt && pip install pyinstaller && PYTHONHASHSEED=31 && pyinstaller -y --specpath installer --add-data /usr/local/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile --name extract_otp_secrets_linux_x86_64_bookworm --distpath /files/dist/ /files/src/extract_otp_secrets.py'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="docker run --entrypoint /bin/bash --rm -v \"$(pwd)\":/files -w /files extract_otp_secrets:buster -c 'apt-get update && apt-get -y install binutils && pip install -U -r /files/requirements.txt && pip install pyinstaller && pyinstaller -y --add-data /usr/local/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile --name extract_otp_secrets_linux_x86_64 --distpath /files/dist/ /files/src/extract_otp_secrets.py'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="dist/extract_otp_secrets_linux_x86_64_bookworm -h || echo 'Could not run exe; see error message above'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="dist/extract_otp_secrets_linux_x86_64 -h"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="dist/extract_otp_secrets_linux_x86_64_bookworm --qr ZBAR example_export.png || echo 'Could not run exe; see error message above'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# create Windows file_version_info.txt
cmd="VERSION_STR=$(setuptools-git-versioning) VERSION_MAJOR=$(cut -d '.' -f 1 <<< "$(setuptools-git-versioning)") VERSION_MINOR=$(cut -d '.' -f 2 <<< "$(setuptools-git-versioning)") VERSION_PATCH=$(cut -d '.' -f 3 <<< "$(setuptools-git-versioning)") VERSION_BUILD=$(($(git rev-list --count $(git tag | sort -V -r | sed '1!d')..HEAD))) YEARS='2020-2023' envsubst < file_version_info_template.txt > build/file_version_info.txt"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Build executable from Docker bullseye
BULLSEYE_GLIBC_VERSION=$(docker run --entrypoint /bin/bash --rm extract_otp_secrets:bullseye -c 'ldd --version | sed "1!d" | sed -E "s/.* ([[:digit:]]+\.[[:digit:]]+)$/\1/"')
echo "Bullseye glibc: $BULLSEYE_GLIBC_VERSION"
cmd="docker run --platform linux/amd64 --entrypoint /bin/bash --rm -v \"$(pwd)\":/files -w /files extract_otp_secrets:bullseye -c 'apt-get update && apt-get -y install binutils && pip install -U pip && pip install -U -r /files/requirements.txt && pip install pyinstaller && PYTHONHASHSEED=31 && pyinstaller -y --specpath installer --add-data /usr/local/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile --name extract_otp_secrets_linux_x86_64 --distpath /files/dist/ /files/src/extract_otp_secrets.py'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="dist/extract_otp_secrets_linux_x86_64 -h"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="dist/extract_otp_secrets_linux_x86_64 --qr ZBAR example_export.png"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
if $build_arm; then
# build linux/arm64
cmd="docker run --platform linux/arm64 --entrypoint /bin/bash --rm -v \"$(pwd)\":/files -w /files extract_otp_secrets:bullseye-arm64 -c 'apt-get update && apt-get -y install binutils && pip install -U pip && pip install -U -r /files/requirements.txt && pip install pyinstaller && PYTHONHASHSEED=31 && pyinstaller --specpath installer -y --add-data /usr/local/__yolo_v3_qr_detector/:__yolo_v3_qr_detector/ --onefile --name extract_otp_secrets_linux_arm64 --distpath /files/dist/ /files/src/extract_otp_secrets.py'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="PLATFORM='linux/arm64' && EXE='dist/extract_otp_secrets_linux_arm64' && docker run --platform \"\$PLATFORM\" --entrypoint /bin/bash --rm -v \"$(pwd)\":/files -w /files extract_otp_secrets:bullseye-arm64 -c \"\$EXE -V && \$EXE -h && \$EXE example_export.png && \$EXE - < example_export.txt && \$EXE --qr ZBAR example_export.png && \$EXE --qr QREADER example_export.png && \$EXE --qr QREADER_DEEP example_export.png && \$EXE --qr CV2 example_export.png && \$EXE --qr CV2_WECHAT example_export.png\""
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
fi
if $build_nuitka_exe; then
if $build_x86_64; then
cmd="docker run --platform linux/amd64 --entrypoint /bin/bash --rm -v \"$(pwd)\":/files -w /files extract_otp_secrets -c 'apt-get update && apt-get -y install binutils build-essential patchelf && pip install -U pip && pip install -U -r /files/requirements.txt && pip install nuitka pyqt5 && PYTHONHASHSEED=31 && python -m nuitka --enable-plugin=tk-inter --enable-plugin=pyqt5 --include-data-dir=/usr/local/__yolo_v3_qr_detector/=__yolo_v3_qr_detector/ --onefile --output-dir=/files/build/docker/nuitka --output-filename=extract_otp_secrets_linux_x86_64_bookworm_compiled /files/src/extract_otp_secrets.py'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="build/docker/nuitka/extract_otp_secrets_linux_x86_64_bookworm_compiled -h"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="build/docker/nuitka/extract_otp_secrets_linux_x86_64_bookworm_compiled --qr ZBAR example_export.png"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="cp build/docker/nuitka/extract_otp_secrets_linux_x86_64_bookworm_compiled dist/"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
# Build executable from Docker bullseye
BULLSEYE_GLIBC_VERSION=$(docker run --entrypoint /bin/bash --rm extract_otp_secrets:bullseye -c 'ldd --version | sed "1!d" | sed -E "s/.* ([[:digit:]]+\.[[:digit:]]+)$/\1/"')
echo "Bookworm glibc: $BULLSEYE_GLIBC_VERSION"
cmd="docker run --platform linux/amd64 --entrypoint /bin/bash --rm -v \"$(pwd)\":/files -w /files extract_otp_secrets:bullseye -c 'apt-get update && apt-get -y install binutils build-essential patchelf && pip install -U pip && pip install -U -r /files/requirements.txt && pip install nuitka pyqt5 && PYTHONHASHSEED=31 && python -m nuitka --enable-plugin=tk-inter --enable-plugin=pyqt5 --include-data-dir=/usr/local/__yolo_v3_qr_detector/=__yolo_v3_qr_detector/ --onefile --output-dir=/files/build/docker/nuitka --output-filename=extract_otp_secrets_linux_x86_64_bullseye_compiled /files/src/extract_otp_secrets.py'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="build/docker/nuitka/extract_otp_secrets_linux_x86_64_bullseye_compiled -h"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="build/docker/nuitka/extract_otp_secrets_linux_x86_64_bullseye_compiled --qr ZBAR example_export.png"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="cp build/docker/nuitka/extract_otp_secrets_linux_x86_64_bullseye_compiled dist/"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
if $build_arm; then
# build linux/arm64
cmd="docker run --platform linux/arm64 --entrypoint /bin/bash --rm -v \"$(pwd)\":/files -w /files extract_otp_secrets:bullseye-arm64 -c 'apt-get update && apt-get -y install binutils build-essential patchelf qt5-default && pip install -U pip && pip install -U -r /files/requirements.txt && pip install nuitka pyqt5 && PYTHONHASHSEED=31 && python -m nuitka --enable-plugin=tk-inter --enable-plugin=pyqt5 --include-data-dir=/usr/local/__yolo_v3_qr_detector/=__yolo_v3_qr_detector/ --onefile --output-dir=/files/build/docker/nuitka --output-filename=extract_otp_secrets_linux_arm64_compiled /files/src/extract_otp_secrets.py'"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="PLATFORM='linux/arm64' && EXE='build/docker/nuitka/extract_otp_secrets_linux_arm64_compiled' && docker run --platform \"\$PLATFORM\" --entrypoint /bin/bash --rm -v \"$(pwd)\":/files -w /files extract_otp_secrets:bullseye-arm64 -c \"\$EXE -V && \$EXE -h && \$EXE example_export.png && \$EXE - < example_export.txt && \$EXE --qr ZBAR example_export.png && \$EXE --qr QREADER example_export.png && \$EXE --qr QREADER_DEEP example_export.png && \$EXE --qr CV2 example_export.png && \$EXE --qr CV2_WECHAT example_export.png\""
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="cp build/docker/nuitka/extract_otp_secrets_linux_arm64_compiled dist/"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
fi
# Run GUI from Docker
if $run_gui; then
if $build_x86_64 && $run_gui; then
cmd="docker run --rm -v "$(pwd)":/files:ro --device=\"/dev/video0:/dev/video0\" --env=\"DISPLAY\" -v /tmp/.X11-unix:/tmp/.X11-unix:ro extract_otp_secrets &"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
@@ -475,14 +658,18 @@ if $run_gui; then
eval "$cmd"
fi
line=$(printf '#%.0s' $(eval echo {1..$(( ($COLUMNS - 10) / 2))}))
echo -e "\n${blueBold}$line RESULTS $line${reset}"
if $build_base; then
line=$(printf '#%.0s' $(eval echo {1..$(( ($COLUMNS - 10) / 2))}))
echo -e "\n${blueBold}$line RESULTS $line${reset}"
cmd="cat $TYPE_CHECK_OUT_FILE $LINT_OUT_FILE $COVERAGE_OUT_FILE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
cmd="cat $TYPE_CHECK_OUT_FILE $LINT_OUT_FILE $COVERAGE_OUT_FILE"
if $interactive ; then askContinueYn "$cmd"; else echo -e "${cyan}$cmd${reset}";fi
eval "$cmd"
fi
line=$(printf '#%.0s' $(eval echo {1..$(( ($COLUMNS - 10) / 2))}))
echo -e "\n${greenBold}$line SUCCESS $line${reset}"
git status
quit

View File

@@ -1,5 +1,5 @@
# --build-arg BASE_IMAGE=python:3.11-slim-buster
ARG BASE_IMAGE=python:3.11-slim-bullseye
ARG BASE_IMAGE=python:3.12-slim-bookworm
FROM $BASE_IMAGE
# https://docs.docker.com/engine/reference/builder/
@@ -16,13 +16,14 @@ COPY requirements*.txt src/ run_pytest.sh pytest.ini tests/ example_*.txt exampl
ARG RUN_TESTS=true
RUN apt-get update && apt-get install -y \
RUN apt-get update && apt-get install -y --no-install-recommends \
libgl1 \
libglib2.0-0 \
libsm6 \
libzbar0 \
python3-tk \
&& rm -rf /var/lib/apt/lists/* \
&& pip install --no-cache-dir -U -r requirements.txt \
&& pip install --no-cache-dir -U pip -r requirements.txt \
&& if [ "$RUN_TESTS" = "true" ]; then /extract/run_pytest.sh; else echo "Not running tests..."; fi \
&& echo 'test -s /extract/.alias && . /extract/.alias || true' >> ~/.bashrc

View File

@@ -1,4 +1,4 @@
ARG BASE_IMAGE=python:3.11-alpine
ARG BASE_IMAGE=python:3.12-alpine
FROM $BASE_IMAGE
# https://docs.docker.com/engine/reference/builder/
@@ -30,6 +30,7 @@ RUN apk add --no-cache \
zlib-dev \
; fi \
&& pip install --no-cache-dir -U \
pip \
colorama \
Pillow \
protobuf \

View File

@@ -3,13 +3,14 @@ Generate from file: README.md
## Table of contents
- [Table of contents](#table-of-contents)
- [Download binary executable (🆕 since v2.1)](#download-binary-executable--since-v21)
- [Download and run binary executable (🆕 since v2.1)](#download-and-run-binary-executable--since-v21)
- [MacOS](#macos)
- [Usage](#usage)
- [Capture QR codes from camera (🆕 since version 2.0)](#capture-qr-codes-from-camera--since-version-20)
- [With builtin QR decoder from image files (🆕 since version 2.0)](#with-builtin-qr-decoder-from-image-files--since-version-20)
- [With external QR decoder app from text files](#with-external-qr-decoder-app-from-text-files)
- [Installation of Python script (recommend for developers or advanced users)](#installation-of-python-script-recommend-for-developers-or-advanced-users)
- [Installation of shared system libraries](#installation-of-shared-system-libraries)
- [Installation of Python script (recommended for developers or advanced users)](#installation-of-python-script-recommended-for-developers-or-advanced-users)
- [Installation of optional shared system libraries (recommended)](#installation-of-optional-shared-system-libraries-recommended)
- [Program help: arguments and options](#program-help-arguments-and-options)
- [Examples](#examples)
- [Printing otp secrets form text file](#printing-otp-secrets-form-text-file)

3
docs/meta.md Normal file
View File

@@ -0,0 +1,3 @@
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=scito/extract_otp_secrets&type=Date)](https://star-history.com/#scito/extract_otp_secrets&Date)

22
installer/build_dmg.sh Executable file
View File

@@ -0,0 +1,22 @@
#!/bin/sh
# Create a folder (named dmg) to prepare our DMG in (if it doesn't already exist).
# https://www.pythonguis.com/tutorials/packaging-pyqt5-applications-pyinstaller-macos-dmg/
mkdir -p dist/dmg
# Empty the dmg folder.
rm -r dist/dmg/*
# Copy the app bundle to the dmg folder.
cp -r "dist/extract_otp_secrets.app" dist/dmg
# If the DMG already exists, delete it.
test -f "dist/extract_otp_secrets.dmg" && rm "dist/extract_otp_secrets.dmg"
create-dmg \
--volname "Extract OTP Secrets" \
--window-pos 200 120 \
--window-size 600 300 \
--icon-size 100 \
--hide-extension "extract_otp_secrets.app" \
--app-drop-link 425 120 \
"dist/extract_otp_secrets.dmg" \
"dist/dmg/"

View File

@@ -0,0 +1,111 @@
# -*- mode: python ; coding: utf-8 -*-
# https://www.pythonguis.com/tutorials/packaging-pyqt5-applications-pyinstaller-macos-dmg/
# https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFBundles/BundleTypes/BundleTypes.html
block_cipher = None
a = Analysis(
['src/extract_otp_secrets.py'],
pathex=[],
binaries=[],
datas=[('$macos_python_path/__yolo_v3_qr_detector/', '__yolo_v3_qr_detector/')],
hiddenimports=[],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
exe = EXE(
pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[],
name='extract_otp_secrets',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
upx_exclude=[],
runtime_tmpdir=None,
console=True,
disable_windowed_traceback=False,
argv_emulation=True,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)
app = BUNDLE(
exe,
name='extract_otp_secrets.app',
icon=None,
bundle_identifier='ch.scito.tools.extract_otp_secrets',
version='$VERSION_STR',
info_plist={
'NSPrincipalClass': 'NSApplication',
'NSAppleScriptEnabled': False,
'NSHumanReadableCopyright': 'Copyright © $COPYRIGHT_YEARS Scito.',
'CFBundleDocumentTypes': [
# Reference: https://chromium.googlesource.com/chromium/src/+/lkgr/chrome/app/app-Info.plist
# https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/understanding_utis/understand_utis_conc/understand_utis_conc.html#//apple_ref/doc/uid/TP40001319-CH202-SW6
# https://developer.apple.com/documentation/uniformtypeidentifiers/system-declared_uniform_type_identifiers
{
'CFBundleTypeName': 'GIF image',
'CFBundleTypeIconFile': 'document.icns',
'LSItemContentTypes': ['com.compuserve.gif'],
'LSHandlerRank': 'Alternate',
'NSExportableTypes': ['public.json','public.comma-separated-values-text','public.plain-text'],
},
{
'CFBundleTypeName': 'JPEG image',
'CFBundleTypeIconFile': 'document.icns',
'LSItemContentTypes': ['public.jpeg'],
'LSHandlerRank': 'Alternate',
'NSExportableTypes': ['public.json','public.comma-separated-values-text','public.plain-text'],
},
{
'CFBundleTypeName': 'PNG image',
'CFBundleTypeIconFile': 'document.icns',
'LSItemContentTypes': ['public.png'],
'LSHandlerRank': 'Alternate',
'NSExportableTypes': ['public.json','public.comma-separated-values-text','public.plain-text'],
},
{
'CFBundleTypeName': 'WebP image',
'CFBundleTypeIconFile': 'document.icns',
'LSItemContentTypes': ['org.webmproject.webp'],
'LSHandlerRank': 'Alternate',
'NSExportableTypes': ['public.json','public.comma-separated-values-text','public.plain-text'],
},
{
'CFBundleTypeName': 'Tiff image',
'CFBundleTypeIconFile': 'document.icns',
'LSItemContentTypes': ['public.tiff'],
'LSHandlerRank': 'Alternate',
'NSExportableTypes': ['public.json','public.comma-separated-values-text','public.plain-text'],
},
{
'CFBundleTypeName': 'Bmp image',
'CFBundleTypeIconFile': 'document.icns',
'LSItemContentTypes': ['com.microsoft.bmp'],
'LSHandlerRank': 'Alternate',
'NSExportableTypes': ['public.json','public.comma-separated-values-text','public.plain-text'],
},
{
'CFBundleTypeName': 'Plain text document',
'CFBundleTypeIconFile': 'document.icns',
'LSItemContentTypes': ['public.plain-text'],
'LSHandlerRank': 'Alternate',
'NSExportableTypes': ['public.json','public.comma-separated-values-text','public.plain-text'],
},
],
},
)

View File

@@ -1,4 +1 @@
[mypy]
[mypy-protobuf_generated_python.*]
ignore_errors = True

View File

@@ -1,8 +1,11 @@
[build-system]
requires = [
"setuptools>=64.0.0", "wheel>=0.37.0", "pip",
"pip",
"nuitka",
# https://setuptools-git-versioning.readthedocs.io/en/latest/differences.html
"setuptools>=64.0.0",
"setuptools-git-versioning",
"wheel>=0.37.0",
]
build-backend = "setuptools.build_meta"
@@ -44,9 +47,6 @@ dependencies = [
"pyzbar",
"qrcode",
"qreader<2.0.0",
# workaround for PYTHON <= 3.7: compatibility
"typing_extensions; python_version<='3.7'",
"importlib_metadata; python_version<='3.7'",
]
description = "Extracts one time password (OTP) secrets from QR codes exported by two-factor authentication (2FA) apps such as 'Google Authenticator'"
dynamic = ["version"]

View File

@@ -3,6 +3,7 @@ flake8
gfm-toc
mypy
mypy-protobuf
nuitka
pyinstaller
pylint
pytest

View File

@@ -1,4 +1,5 @@
colorama>=0.4.6
importlib_metadata; python_version<='3.7'
opencv-contrib-python; sys_platform != 'darwin'
opencv-contrib-python<=4.7.0; sys_platform == 'darwin'
Pillow
@@ -7,4 +8,3 @@ pyzbar
qrcode
qreader<2.0.0
typing_extensions; python_version<='3.7'
importlib_metadata; python_version<='3.7'

View File

@@ -43,43 +43,44 @@ import re
import sys
import urllib.parse as urlparse
from enum import Enum, IntEnum
from typing import Any, List, Optional, Sequence, TextIO, Tuple, Union
from importlib.metadata import PackageNotFoundError, version
from typing import (Any, Final, List, Optional, Sequence, TextIO, Tuple,
TypedDict, Union)
import colorama
from pkg_resources import DistributionNotFound, get_distribution
from qrcode import QRCode # type: ignore
import protobuf_generated_python.google_auth_pb2 as pb
# workaround for PYTHON <= 3.7: compatibility
if sys.version_info >= (3, 8):
from typing import Final, TypedDict
else:
from typing_extensions import Final, TypedDict
# workaround for PYTHON <= 3.7: compatibility
if sys.version_info >= (3, 8):
from importlib.metadata import PackageNotFoundError, version
else:
from importlib_metadata import PackageNotFoundError, version
debug_mode = '-d' in sys.argv[1:] or '--debug' in sys.argv[1:]
quiet = '-q' in sys.argv[1:] or '--quiet' in sys.argv[1:]
headless: bool = False
try:
import cv2 # type: ignore # TODO use cv2 types if available
import numpy as np # TODO use numpy types if available
import cv2
import numpy as np
try:
import tkinter
import tkinter.filedialog
import tkinter.messagebox
except ImportError:
headless = True
try:
import pyzbar.pyzbar as zbar # type: ignore
from qreader import QReader # type: ignore
except ImportError as e:
print(f"""
ERROR: Cannot import QReader module. This problem is probably due to the missing zbar shared library.
On Linux and macOS libzbar0 must be installed.
See in README.md for the installation of the libzbar0.
zbar_available = True
except Exception as e:
if not quiet:
print(f"""
WARN: Cannot import pyzbar module. This problem is probably due to the missing zbar shared library. (The zbar library is optional.)
See in README.md for the installation of the zbar shared library.
Exception: {e}\n""", file=sys.stderr)
raise e
zbar_available = False
if debug_mode:
raise e
# Types
# workaround for PYTHON <= 3.9: Final[tuple[int]]
@@ -91,9 +92,8 @@ Exception: {e}\n""", file=sys.stderr)
FONT_SCALE: Final[float] = 1.3
FONT_THICKNESS: Final[int] = 1
FONT_LINE_STYLE: Final[int] = cv2.LINE_AA
FONT_COLOR: Final[ColorBGR] = (255, 0, 0)
FONT_COLOR: Final[ColorBGR] = 255, 0, 0
BOX_THICKNESS: Final[int] = 5
# workaround for PYTHON <= 3.7: must use () for assignments
WINDOW_X: Final[int] = 0
WINDOW_Y: Final[int] = 1
WINDOW_WIDTH: Final[int] = 2
@@ -102,28 +102,28 @@ Exception: {e}\n""", file=sys.stderr)
TEXT_HEIGHT: Final[int] = 1
BORDER: Final[int] = 5
START_Y: Final[int] = 20
START_POS_TEXT: Final[Point] = (BORDER, START_Y)
NORMAL_COLOR: Final[ColorBGR] = (255, 0, 255)
SUCCESS_COLOR: Final[ColorBGR] = (0, 255, 0)
FAILURE_COLOR: Final[ColorBGR] = (0, 0, 255)
START_POS_TEXT: Final[Point] = BORDER, START_Y
NORMAL_COLOR: Final[ColorBGR] = 255, 0, 255
SUCCESS_COLOR: Final[ColorBGR] = 0, 255, 0
FAILURE_COLOR: Final[ColorBGR] = 0, 0, 255
CHAR_DX: Final[int] = (lambda text: cv2.getTextSize(text, FONT, FONT_SCALE, FONT_THICKNESS)[0][TEXT_WIDTH] // len(text))("28 QR codes capturedMMM")
FONT_DY: Final[int] = cv2.getTextSize("M", FONT, FONT_SCALE, FONT_THICKNESS)[0][TEXT_HEIGHT] + 5
WINDOW_NAME: Final[str] = "Extract OTP Secrets: Capture QR Codes from Camera"
TextPosition = Enum('TextPosition', ['LEFT', 'RIGHT'])
qreader_available = True
cv2_available = True
except ImportError as e:
qreader_available = False
cv2_available = False
if debug_mode:
raise e
# Workaround for PYTHON <= 3.9: Union[int, None] used instead of int | None
# Workaround for PYTHON <= 3.9: Generally Union[int, None] used instead of int | None
# Types
Args = argparse.Namespace
OtpUrl = str
# workaround for PYTHON <= 3.7: Otp = TypedDict('Otp', {'name': str, 'secret': str, 'issuer': str, 'type': str, 'counter': int | None, 'url': OtpUrl})
# Workaround for PYTHON <= 3.9: Otp = TypedDict('Otp', {'name': str, 'secret': str, 'issuer': str, 'type': str, 'counter': int | None, 'url': OtpUrl})
Otp = TypedDict('Otp', {'name': str, 'secret': str, 'issuer': str, 'type': str, 'counter': Union[int, None], 'url': OtpUrl})
# workaround for PYTHON <= 3.9: Otps = list[Otp]
Otps = List[Otp]
@@ -136,13 +136,14 @@ LogLevel = IntEnum('LogLevel', ['QUIET', 'NORMAL', 'VERBOSE', 'MORE_VERBOSE', 'D
# Constants
CAMERA: Final[str] = 'camera'
CV2_QRMODES: List[str] = [QRMode.CV2.name, QRMode.CV2_WECHAT.name]
# Global variable declaration
verbose: IntEnum = LogLevel.NORMAL
quiet: bool = False
colored: bool = True
executable: bool = False
__version__: str
tk_root: tkinter.Tk
def sys_main() -> None:
@@ -150,7 +151,7 @@ def sys_main() -> None:
def main(sys_args: list[str]) -> None:
global executable
global executable, tk_root, headless
# allow to use sys.stdout with with (avoid closing)
sys.stdout.close = lambda: None # type: ignore
# set encoding to utf-8, needed for Windows
@@ -164,6 +165,13 @@ def main(sys_args: list[str]) -> None:
# https://pyinstaller.org/en/stable/runtime-information.html#run-time-information
executable = getattr(sys, 'frozen', False) and hasattr(sys, '_MEIPASS')
if cv2_available and not headless:
try:
tk_root = tkinter.Tk()
tk_root.withdraw()
except tkinter.TclError:
headless = True
args = parse_args(sys_args)
if colored:
@@ -174,9 +182,11 @@ def main(sys_args: list[str]) -> None:
sys.exit(0 if do_debug_checks() else 1)
otps = extract_otps(args)
write_csv(args, otps)
write_keepass_csv(args, otps)
write_json(args, otps)
write_csv(args.csv, otps)
write_keepass_csv(args.keepass, otps)
write_json(args.json, otps)
write_txt(args.txt, otps, True)
# workaround for PYTHON <= 3.9 use: pb.MigrationPayload | None
@@ -239,9 +249,9 @@ def extract_otp_from_otp_url(otpauth_migration_url: str, otps: Otps, urls_count:
if not quiet:
print_otp(otp)
if args.printqr:
print_qr(args, otp_url)
print_qr(otp_url)
if args.saveqr:
save_qr(otp, args, len(otps))
save_qr_image(otp, args.saveqr, len(otps))
if not quiet:
print()
elif args.ignore and not quiet:
@@ -253,11 +263,9 @@ def extract_otp_from_otp_url(otpauth_migration_url: str, otps: Otps, urls_count:
def parse_args(sys_args: list[str]) -> Args:
global verbose, quiet, colored
# For PYTHON <= 3.7: Use :=
name = os.path.basename(sys.argv[0])
cmd = f"python {name}" if name.endswith('.py') else f"{name}"
cmd = f"python {name}" if (name := os.path.basename(sys.argv[0])).endswith('.py') else f"{name}"
description_text = "Extracts one time password (OTP) secrets from QR codes exported by two-factor authentication (2FA) apps"
if qreader_available:
if cv2_available:
description_text += "\nIf no infiles are provided, a GUI window starts and QR codes are captured from the camera."
example_text = f"""examples:
{cmd}
@@ -270,15 +278,19 @@ def parse_args(sys_args: list[str]) -> Args:
description=description_text,
epilog=example_text)
arg_parser.add_argument('infile', help="""a) file or - for stdin with 'otpauth-migration://...' URLs separated by newlines, lines starting with # are ignored;
b) image file containing a QR code or = for stdin for an image containing a QR code""", nargs='*' if qreader_available else '+')
b) image file containing a QR code or = for stdin for an image containing a QR code""", nargs='*' if cv2_available else '+')
arg_parser.add_argument('--csv', '-c', help='export csv file or - for stdout', metavar=('FILE'))
arg_parser.add_argument('--keepass', '-k', help='export totp/hotp csv file(s) for KeePass, - for stdout', metavar=('FILE'))
arg_parser.add_argument('--json', '-j', help='export json file or - for stdout', metavar=('FILE'))
arg_parser.add_argument('--printqr', '-p', help='print QR code(s) as text to the terminal (requires qrcode module)', action='store_true')
arg_parser.add_argument('--saveqr', '-s', help='save QR code(s) as images to the given folder (requires qrcode module)', metavar=('DIR'))
if qreader_available:
arg_parser.add_argument('--txt', '-t', help='export txt file or - for stdout', metavar=('FILE'))
arg_parser.add_argument('--printqr', '-p', help='print QR code(s) as text to the terminal', action='store_true')
arg_parser.add_argument('--saveqr', '-s', help='save QR code(s) as images to directory', metavar=('DIR'))
if cv2_available:
arg_parser.add_argument('--camera', '-C', help='camera number of system (default camera: 0)', default=0, type=int, metavar=('NUMBER'))
arg_parser.add_argument('--qr', '-Q', help=f'QR reader (default: {QRMode.ZBAR.name})', type=str, choices=[mode.name for mode in QRMode], default=QRMode.ZBAR.name)
if not zbar_available:
arg_parser.add_argument('--qr', '-Q', help=f'QR reader (default: {QRMode.CV2.name})', type=str, choices=[QRMode.CV2.name, QRMode.CV2_WECHAT.name], default=QRMode.CV2.name)
else:
arg_parser.add_argument('--qr', '-Q', help=f'QR reader (default: {QRMode.ZBAR.name})', type=str, choices=[mode.name for mode in QRMode], default=QRMode.ZBAR.name)
arg_parser.add_argument('-i', '--ignore', help='ignore duplicate otps', action='store_true')
arg_parser.add_argument('--no-color', '-n', help='do not use ANSI colors in console output', action='store_true')
arg_parser.add_argument('--version', '-V', help='print version and quit', action=PrintVersionAction)
@@ -288,7 +300,7 @@ b) image file containing a QR code or = for stdin for an image containing a QR c
output_group.add_argument('-q', '--quiet', help='no stdout output, except output set by -', action='store_true')
args = arg_parser.parse_args(sys_args)
colored = not args.no_color
if args.csv == '-' or args.json == '-' or args.keepass == '-':
if args.csv == '-' or args.json == '-' or args.keepass == '-' or args.txt == '-':
args.quiet = args.q = True
verbose = args.verbose if args.verbose else LogLevel.NORMAL
@@ -296,9 +308,9 @@ b) image file containing a QR code or = for stdin for an image containing a QR c
verbose = LogLevel.DEBUG
log_debug('Debug mode start')
quiet = True if args.quiet else False
if verbose: print(f"QReader installed: {qreader_available}")
if qreader_available:
if verbose >= LogLevel.VERBOSE: print(f"CV2 version: {cv2.__version__}")
if verbose: print(f"QReader installed: {cv2_available}")
if cv2_available:
if verbose >= LogLevel.VERBOSE: print(f"CV2 version: {cv2.__version__}") # type: ignore # cv2.__version__ is not available
if verbose: print(f"QR reading mode: {args.qr}\n")
return args
@@ -321,7 +333,8 @@ def extract_otps_from_camera(args: Args) -> Otps:
cam = cv2.VideoCapture(args.camera)
cv2.namedWindow(WINDOW_NAME, cv2.WINDOW_AUTOSIZE)
qreader = QReader()
if zbar_available:
qreader = QReader()
cv2_qr = cv2.QRCodeDetector()
cv2_qr_wechat = cv2.wechat_qrcode.WeChatQRCode()
while True:
@@ -362,15 +375,16 @@ def extract_otps_from_camera(args: Args) -> Otps:
qr_mode = next_qr_mode(qr_mode)
continue
cv2_print_text(img, f"Mode: {qr_mode.name} (Hit space to change)", 0, TextPosition.LEFT, FONT_COLOR, 20)
cv2_print_text(img, "Hit ESC to quit", 1, TextPosition.LEFT, FONT_COLOR, 17)
cv2_print_text(img, f"Mode: {qr_mode.name} (Hit SPACE to change)", 0, TextPosition.LEFT, FONT_COLOR, 20)
cv2_print_text(img, "Press ESC to quit", 1, TextPosition.LEFT, FONT_COLOR, 17)
cv2_print_text(img, "Press C/J/K/T to save as csv/json/keepass/txt file", 2, TextPosition.LEFT, FONT_COLOR, None)
cv2_print_text(img, f"{len(otp_urls)} QR code{'s'[:len(otp_urls) != 1]} captured", 0, TextPosition.RIGHT, FONT_COLOR)
cv2_print_text(img, f"{len(otps)} otp{'s'[:len(otps) != 1]} extracted", 1, TextPosition.RIGHT, FONT_COLOR)
cv2.imshow(WINDOW_NAME, img)
quit, qr_mode = cv2_handle_pressed_keys(qr_mode)
quit, qr_mode = cv2_handle_pressed_keys(qr_mode, otps)
if quit:
break
@@ -391,7 +405,7 @@ def get_color(new_otps_count: int, otp_url: str) -> ColorBGR:
# TODO use cv2 types if available
def cv2_draw_box(img: Any, raw_pts: Any, color: ColorBGR) -> Any:
def cv2_draw_box(img: list[tuple[Any, Any]], raw_pts: list[tuple[Any, Any]], color: ColorBGR) -> np.ndarray[Any, np.dtype[np.int32]]:
pts = np.array([raw_pts], np.int32)
pts = pts.reshape((-1, 1, 2))
cv2.polylines(img, [pts], True, color, BOX_THICKNESS)
@@ -399,7 +413,7 @@ def cv2_draw_box(img: Any, raw_pts: Any, color: ColorBGR) -> Any:
# TODO use cv2 types if available
def cv2_print_text(img: Any, text: str, line_number: int, position: TextPosition, color: ColorBGR, opposite_len: Optional[int] = None) -> None:
def cv2_print_text(img: cv2.UMat, text: str, line_number: int, position: TextPosition, color: ColorBGR, opposite_len: Optional[int] = None) -> None:
window_dim = cv2.getWindowImageRect(WINDOW_NAME)
out_text = text
if opposite_len:
@@ -416,14 +430,62 @@ def cv2_print_text(img: Any, text: str, line_number: int, position: TextPosition
cv2.putText(img, out_text, pos, FONT, FONT_SCALE, color, FONT_THICKNESS, FONT_LINE_STYLE)
def cv2_handle_pressed_keys(qr_mode: QRMode) -> Tuple[bool, QRMode]:
def cv2_handle_pressed_keys(qr_mode: QRMode, otps: Otps) -> Tuple[bool, QRMode]:
key = cv2.waitKey(1) & 0xFF
quit = False
if key == 27 or key == ord('q') or key == 13:
if key == 27 or key == ord('q') or key == ord('Q') or key == 13:
# ESC or Enter or q pressed
quit = True
elif (key == ord('c') or key == ord('C')) and is_not_headless():
if has_no_otps_show_warning(otps):
pass
else:
file_name = tkinter.filedialog.asksaveasfilename(
title="Save extracted otp secrets as CSV",
defaultextension='.csv',
filetypes=[('CSV', '*.csv'), ('All', '*.*')]
)
tk_root.update()
if len(file_name) > 0:
write_csv(file_name, otps)
elif (key == ord('j') or key == ord('J')) and is_not_headless():
if has_no_otps_show_warning(otps):
pass
else:
file_name = tkinter.filedialog.asksaveasfilename(
title="Save extracted otp secrets as JSON",
defaultextension='.json',
filetypes=[('JSON', '*.json'), ('All', '*.*')]
)
tk_root.update()
if len(file_name) > 0:
write_json(file_name, otps)
elif (key == ord('k') or key == ord('K')) and is_not_headless():
if has_no_otps_show_warning(otps):
pass
else:
file_name = tkinter.filedialog.asksaveasfilename(
title="Save extracted otp secrets as KeePass CSV file(s)",
defaultextension='.csv',
filetypes=[('CSV', '*.csv'), ('All', '*.*')]
)
tk_root.update()
if len(file_name) > 0:
write_keepass_csv(file_name, otps)
elif (key == ord('t') or key == ord('T')) and is_not_headless():
if has_no_otps_show_warning(otps):
pass
else:
file_name = tkinter.filedialog.asksaveasfilename(
title="Save extracted otp secrets as text",
defaultextension='.txt',
filetypes=[('Text', '*.txt'), ('All', '*.*')]
)
tk_root.update()
if len(file_name) > 0:
write_txt(file_name, otps, True)
elif key == 32:
qr_mode = next_qr_mode(qr_mode)
qr_mode = next_valid_qr_mode(qr_mode, zbar_available)
if verbose >= LogLevel.MORE_VERBOSE: print(f"QR reading mode: {qr_mode}")
if cv2.getWindowProperty(WINDOW_NAME, cv2.WND_PROP_VISIBLE) < 1:
# Window close clicked
@@ -471,7 +533,7 @@ def get_otp_urls_from_file(filename: str, args: Args) -> OtpUrls:
return lines
# could not process text file, try reading as image
if filename != '-' and qreader_available:
if filename != '-' and cv2_available:
return convert_img_to_otp_urls(filename, args)
return []
@@ -591,28 +653,27 @@ def build_otp_url(secret: str, raw_otp: pb.MigrationPayload.OtpParameters) -> st
return otp_url
def print_otp(otp: Otp) -> None:
print(f"Name: {otp['name']}")
print(f"Secret: {otp['secret']}")
if otp['issuer']: print(f"Issuer: {otp['issuer']}")
print(f"Type: {otp['type']}")
def print_otp(otp: Otp, out: Optional[TextIO] = None) -> None:
print(f"Name: {otp['name']}", file=out)
print(f"Secret: {otp['secret']}", file=out)
if otp['issuer']: print(f"Issuer: {otp['issuer']}", file=out)
print(f"Type: {otp['type']}", file=out)
if otp['type'] == 'hotp':
print(f"Counter: {otp['counter']}")
print(f"Counter: {otp['counter']}", file=out)
if verbose:
print(otp['url'])
print(otp['url'], file=out)
def save_qr(otp: Otp, args: Args, j: int) -> str:
dir = args.saveqr
def save_qr_image(otp: Otp, dir: str, j: int) -> str:
if not (os.path.exists(dir)): os.makedirs(dir, exist_ok=True)
pattern = re.compile(r'[\W_]+')
file_otp_name = pattern.sub('', otp['name'])
file_otp_issuer = pattern.sub('', otp['issuer'])
save_qr_file(args, otp['url'], f"{dir}/{j}-{file_otp_name}{'-' + file_otp_issuer if file_otp_issuer else ''}.png")
save_qr_image_file(otp['url'], f"{dir}/{j}-{file_otp_name}{'-' + file_otp_issuer if file_otp_issuer else ''}.png")
return file_otp_name
def save_qr_file(args: Args, otp_url: OtpUrl, name: str) -> None:
def save_qr_image_file(otp_url: OtpUrl, name: str) -> None:
qr = QRCode()
qr.add_data(otp_url)
img = qr.make_image(fill_color='black', back_color='white')
@@ -620,28 +681,38 @@ def save_qr_file(args: Args, otp_url: OtpUrl, name: str) -> None:
img.save(name)
def print_qr(args: Args, otp_url: str) -> None:
def print_qr(otp_url: str, out: Optional[TextIO] = None) -> None:
qr = QRCode()
qr.add_data(otp_url)
qr.print_ascii()
qr.print_ascii(out)
def write_csv(args: Args, otps: Otps) -> None:
if args.csv and len(otps) > 0:
with open_file_or_stdout_for_csv(args.csv) as outfile:
def write_txt(file: str, otps: Otps, write_qr: bool = False) -> None:
if file and len(file) > 0 and len(otps) > 0:
with open_file_or_stdout(file) as outfile:
for otp in otps:
print_otp(otp, outfile)
if write_qr:
print_qr(otp['url'], outfile)
print(file=outfile)
def write_csv(file: str, otps: Otps) -> None:
if file and len(file) > 0 and len(otps) > 0:
with open_file_or_stdout_for_csv(file) as outfile:
writer = csv.DictWriter(outfile, otps[0].keys())
writer.writeheader()
writer.writerows(otps)
if not quiet: print(f"Exported {len(otps)} otp{'s'[:len(otps) != 1]} to csv {args.csv}")
if not quiet: print(f"Exported {len(otps)} otp{'s'[:len(otps) != 1]} to csv {file}")
def write_keepass_csv(args: Args, otps: Otps) -> None:
if args.keepass and len(otps) > 0:
def write_keepass_csv(file: str, otps: Otps) -> None:
if file and len(file) > 0 and len(otps) > 0:
has_totp = has_otp_type(otps, 'totp')
has_hotp = has_otp_type(otps, 'hotp')
if args.keepass != '-':
otp_filename_totp = args.keepass if has_totp != has_hotp else add_pre_suffix(args.keepass, "totp")
otp_filename_hotp = args.keepass if has_totp != has_hotp else add_pre_suffix(args.keepass, "hotp")
if file != '-':
otp_filename_totp = file if has_totp != has_hotp else add_pre_suffix(file, "totp")
otp_filename_hotp = file if has_totp != has_hotp else add_pre_suffix(file, "hotp")
else:
otp_filename_totp = otp_filename_hotp = '-'
if has_totp:
@@ -649,13 +720,13 @@ def write_keepass_csv(args: Args, otps: Otps) -> None:
if has_hotp:
count_hotp_entries = write_keepass_htop_csv(otp_filename_hotp, otps)
if not quiet:
if count_totp_entries: print(f"Exported {count_totp_entries} totp entrie{'s'[:count_totp_entries != 1]} to keepass csv file {otp_filename_totp}")
if count_hotp_entries: print(f"Exported {count_hotp_entries} hotp entrie{'s'[:count_hotp_entries != 1]} to keepass csv file {otp_filename_hotp}")
if has_totp and count_totp_entries: print(f"Exported {count_totp_entries} totp entrie{'s'[:count_totp_entries != 1]} to keepass csv file {otp_filename_totp}")
if has_hotp and count_hotp_entries: print(f"Exported {count_hotp_entries} hotp entrie{'s'[:count_hotp_entries != 1]} to keepass csv file {otp_filename_hotp}")
def write_keepass_totp_csv(otp_filename: str, otps: Otps) -> int:
def write_keepass_totp_csv(file: str, otps: Otps) -> int:
count_entries = 0
with open_file_or_stdout_for_csv(otp_filename) as outfile:
with open_file_or_stdout_for_csv(file) as outfile:
writer = csv.DictWriter(outfile, ["Title", "User Name", "TimeOtp-Secret-Base32", "Group"])
writer.writeheader()
for otp in otps:
@@ -670,9 +741,9 @@ def write_keepass_totp_csv(otp_filename: str, otps: Otps) -> int:
return count_entries
def write_keepass_htop_csv(otp_filename: str, otps: Otps) -> int:
def write_keepass_htop_csv(file: str, otps: Otps) -> int:
count_entries = 0
with open_file_or_stdout_for_csv(otp_filename) as outfile:
with open_file_or_stdout_for_csv(file) as outfile:
writer = csv.DictWriter(outfile, ["Title", "User Name", "HmacOtp-Secret-Base32", "HmacOtp-Counter", "Group"])
writer.writeheader()
for otp in otps:
@@ -688,11 +759,11 @@ def write_keepass_htop_csv(otp_filename: str, otps: Otps) -> int:
return count_entries
def write_json(args: Args, otps: Otps) -> None:
if args.json:
with open_file_or_stdout(args.json) as outfile:
def write_json(file: str, otps: Otps) -> None:
if file and len(file) > 0:
with open_file_or_stdout(file) as outfile:
json.dump(otps, outfile, indent=4)
if not quiet: print(f"Exported {len(otps)} otp{'s'[:len(otps) != 1]} to json {args.json}")
if not quiet: print(f"Exported {len(otps)} otp{'s'[:len(otps) != 1]} to json {file}")
def has_otp_type(otps: Otps, otp_type: str) -> bool:
@@ -729,6 +800,13 @@ def check_file_exists(filename: str) -> None:
f"\ninput file: {filename}")
def has_no_otps_show_warning(otps: Otps) -> bool:
if len(otps) == 0:
tkinter.messagebox.showinfo(title="No data", message="There are no otp secrets to write")
tk_root.update() # dispose dialog
return len(otps) == 0
def is_binary(line: str) -> bool:
try:
line.startswith('#')
@@ -737,6 +815,14 @@ def is_binary(line: str) -> bool:
return True
def next_valid_qr_mode(qr_mode: QRMode, with_zbar: bool = True) -> QRMode:
ok = False
while not ok:
qr_mode = next_qr_mode(qr_mode)
ok = True if with_zbar else qr_mode.name in CV2_QRMODES
return qr_mode
def next_qr_mode(qr_mode: QRMode) -> QRMode:
return QRMode((qr_mode.value + 1) % len(QRMode))
@@ -747,10 +833,20 @@ def do_debug_checks() -> bool:
import cv2 # noqa: F401 # This is only a debug import
log_debug('Try: import numpy as np')
import numpy as np # noqa: F401 # This is only a debug import
log_debug('Try: import pyzbar.pyzbar as zbar')
import pyzbar.pyzbar as zbar # noqa: F401 # This is only a debug import
log_debug('Try: from qreader import QReader')
from qreader import QReader # noqa: F401 # This is only a debug import
print(color('\nDebug checks passed', colorama.Fore.GREEN))
return True
def is_not_headless() -> bool:
if headless:
log_warn("Cannot open dialog in headless mode")
return not headless
class PrintVersionAction(argparse.Action):
def __init__(self, option_strings: Sequence[str], dest: str, nargs: int = 0, **kwargs: Any) -> None:
super().__init__(option_strings, dest, nargs, **kwargs)
@@ -787,19 +883,7 @@ def get_raw_version() -> str:
return __version__
except PackageNotFoundError:
# package is not installed
pass
# In some cases importlib cannot properly detect package version, for example it was compiled into executable file, so it uses some custom import mechanism.
# Instead, use pkg_resources which is included in setuptools (but has a significant runtime cost)
try:
__version__ = get_distribution("package-name").version
return __version__
except DistributionNotFound:
# package is not installed
pass
return ''
return ''
# workaround for PYTHON <= 3.9 use: BaseException | None

View File

@@ -2,10 +2,10 @@
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: google_auth.proto
"""Generated protocol buffer code."""
from google.protobuf.internal import builder as _builder
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
@@ -15,17 +15,17 @@ _sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x11google_auth.proto\"\xb7\x03\n\x10MigrationPayload\x12\x37\n\x0eotp_parameters\x18\x01 \x03(\x0b\x32\x1f.MigrationPayload.OtpParameters\x12\x0f\n\x07version\x18\x02 \x01(\x05\x12\x12\n\nbatch_size\x18\x03 \x01(\x05\x12\x13\n\x0b\x62\x61tch_index\x18\x04 \x01(\x05\x12\x10\n\x08\x62\x61tch_id\x18\x05 \x01(\x05\x1a\xb7\x01\n\rOtpParameters\x12\x0e\n\x06secret\x18\x01 \x01(\x0c\x12\x0c\n\x04name\x18\x02 \x01(\t\x12\x0e\n\x06issuer\x18\x03 \x01(\t\x12.\n\talgorithm\x18\x04 \x01(\x0e\x32\x1b.MigrationPayload.Algorithm\x12\x0e\n\x06\x64igits\x18\x05 \x01(\x05\x12\'\n\x04type\x18\x06 \x01(\x0e\x32\x19.MigrationPayload.OtpType\x12\x0f\n\x07\x63ounter\x18\x07 \x01(\x03\",\n\tAlgorithm\x12\x10\n\x0c\x41LGO_INVALID\x10\x00\x12\r\n\tALGO_SHA1\x10\x01\"6\n\x07OtpType\x12\x0f\n\x0bOTP_INVALID\x10\x00\x12\x0c\n\x08OTP_HOTP\x10\x01\x12\x0c\n\x08OTP_TOTP\x10\x02\x62\x06proto3')
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google_auth_pb2', globals())
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google_auth_pb2', _globals)
if _descriptor._USE_C_DESCRIPTORS == False:
DESCRIPTOR._options = None
_MIGRATIONPAYLOAD._serialized_start=22
_MIGRATIONPAYLOAD._serialized_end=461
_MIGRATIONPAYLOAD_OTPPARAMETERS._serialized_start=176
_MIGRATIONPAYLOAD_OTPPARAMETERS._serialized_end=359
_MIGRATIONPAYLOAD_ALGORITHM._serialized_start=361
_MIGRATIONPAYLOAD_ALGORITHM._serialized_end=405
_MIGRATIONPAYLOAD_OTPTYPE._serialized_start=407
_MIGRATIONPAYLOAD_OTPTYPE._serialized_end=461
_globals['_MIGRATIONPAYLOAD']._serialized_start=22
_globals['_MIGRATIONPAYLOAD']._serialized_end=461
_globals['_MIGRATIONPAYLOAD_OTPPARAMETERS']._serialized_start=176
_globals['_MIGRATIONPAYLOAD_OTPPARAMETERS']._serialized_end=359
_globals['_MIGRATIONPAYLOAD_ALGORITHM']._serialized_start=361
_globals['_MIGRATIONPAYLOAD_ALGORITHM']._serialized_end=405
_globals['_MIGRATIONPAYLOAD_OTPTYPE']._serialized_start=407
_globals['_MIGRATIONPAYLOAD_OTPTYPE']._serialized_end=461
# @@protoc_insertion_point(module_scope)

View File

@@ -28,7 +28,7 @@ class MigrationPayload(google.protobuf.message.Message):
ValueType = typing.NewType("ValueType", builtins.int)
V: typing_extensions.TypeAlias = ValueType
class _AlgorithmEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[MigrationPayload._Algorithm.ValueType], builtins.type): # noqa: F821
class _AlgorithmEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[MigrationPayload._Algorithm.ValueType], builtins.type):
DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor
ALGO_INVALID: MigrationPayload._Algorithm.ValueType # 0
ALGO_SHA1: MigrationPayload._Algorithm.ValueType # 1
@@ -41,7 +41,7 @@ class MigrationPayload(google.protobuf.message.Message):
ValueType = typing.NewType("ValueType", builtins.int)
V: typing_extensions.TypeAlias = ValueType
class _OtpTypeEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[MigrationPayload._OtpType.ValueType], builtins.type): # noqa: F821
class _OtpTypeEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[MigrationPayload._OtpType.ValueType], builtins.type):
DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor
OTP_INVALID: MigrationPayload._OtpType.ValueType # 0
OTP_HOTP: MigrationPayload._OtpType.ValueType # 1

View File

@@ -39,17 +39,17 @@ from utils import (count_files_in_dir, file_exits, read_binary_file_as_stream,
import extract_otp_secrets
try:
import cv2 # type: ignore
import cv2
from extract_otp_secrets import SUCCESS_COLOR, FAILURE_COLOR, FONT, FONT_SCALE, FONT_COLOR, FONT_THICKNESS, FONT_LINE_STYLE
except ImportError:
# ignore
pass
qreader_available: bool = extract_otp_secrets.qreader_available
cv2_available: bool = extract_otp_secrets.cv2_available
# Quickfix comment
# @pytest.mark.skipif(sys.platform.startswith("win") or not qreader_available or sys.implementation.name == 'pypy' or sys.version_info >= (3, 10), reason="Quickfix")
# @pytest.mark.skipif(sys.platform.startswith("win") or not cv2 or sys.implementation.name == 'pypy' or sys.version_info >= (3, 10), reason="Quickfix")
def test_extract_stdout(capsys: pytest.CaptureFixture[str]) -> None:
@@ -122,7 +122,7 @@ def test_extract_stdin_only_comments(capsys: pytest.CaptureFixture[str], monkeyp
def test_extract_empty_file_no_qreader(capsys: pytest.CaptureFixture[str]) -> None:
if qreader_available:
if cv2_available:
# Act
with pytest.raises(SystemExit) as e:
extract_otp_secrets.main(['-n', 'tests/data/empty_file.txt'])
@@ -376,6 +376,48 @@ def test_extract_json_stdout_only_comments(capsys: pytest.CaptureFixture[str]) -
assert captured.err == ''
def test_extract_txt(capsys: pytest.CaptureFixture[str], tmp_path: pathlib.Path) -> None:
# Arrange
output_file = str(tmp_path / 'test_example_output.txt')
# Act
extract_otp_secrets.main(['-q', '-t', output_file, 'example_export.txt'])
# Assert
expected_txt = read_file_to_str('tests/data/printqr_output.txt')
actual_txt = read_file_to_str(output_file)
assert actual_txt == expected_txt
captured = capsys.readouterr()
assert captured.out == ''
assert captured.err == ''
def test_extract_txt_stdout(capsys: pytest.CaptureFixture[str]) -> None:
# Act
extract_otp_secrets.main(['-t', '-', 'example_export.txt'])
# Assert
expected_txt = read_file_to_str('tests/data/printqr_output.txt')
captured = capsys.readouterr()
assert captured.out == expected_txt
assert captured.err == ''
def test_extract_txt_stdout_only_comments(capsys: pytest.CaptureFixture[str]) -> None:
# Act
extract_otp_secrets.main(['-t', '-', 'tests/data/only_comments.txt'])
# Assert
captured = capsys.readouterr()
assert captured.out == ''
assert captured.err == ''
def test_extract_not_encoded_plus(capsys: pytest.CaptureFixture[str]) -> None:
# Act
extract_otp_secrets.main(['tests/data/test_plus_problem_export.txt'])
@@ -510,7 +552,7 @@ def test_extract_verbose(verbose_level: str, color: str, capsys: pytest.CaptureF
def normalize_verbose_text(text: str, relaxed: bool) -> str:
normalized = re.sub('^.*version: .+$', '', text, flags=re.MULTILINE | re.IGNORECASE)
if not qreader_available:
if not cv2_available:
normalized = normalized \
.replace('QReader installed: True', 'QReader installed: False') \
.replace('\nQR reading mode: ZBAR\n\n', '')
@@ -564,7 +606,7 @@ def test_extract_version(capsys: pytest.CaptureFixture[str]) -> None:
def test_extract_no_arguments(capsys: pytest.CaptureFixture[str], mocker: MockerFixture) -> None:
if qreader_available:
if cv2_available:
# Arrange
otps = read_json('example_output.json')
mocker.patch('extract_otp_secrets.extract_otps_from_camera', return_value=otps)
@@ -648,7 +690,7 @@ class MockCam:
('CV2_WECHAT', 'tests/data/lena_std.tif', None),
])
def test_extract_otps_from_camera(qr_reader: Optional[str], file: str, success: bool, capsys: pytest.CaptureFixture[str], mocker: MockerFixture) -> None:
if qreader_available:
if cv2_available:
# Arrange
mockCam = MockCam([file])
mocker.patch('cv2.VideoCapture', return_value=mockCam)
@@ -724,8 +766,10 @@ def test_verbose_and_quiet(capsys: pytest.CaptureFixture[str]) -> None:
('-k', 'outfile', False, False),
('-k', '-', True, False),
('-j', 'outfile', False, False),
('-s', 'outfile', False, False),
('-j', '-', True, False),
('-t', 'outfile', False, False),
('-t', '-', True, False),
('-s', 'outfile', False, False),
('-i', None, False, False),
('-p', None, True, False),
('-Q', 'CV2', False, False),
@@ -733,7 +777,7 @@ def test_verbose_and_quiet(capsys: pytest.CaptureFixture[str]) -> None:
('-n', None, False, False),
])
def test_quiet(parameter: str, parameter_value: Optional[str], stdout_expected: bool, stderr_expected: bool, capsys: pytest.CaptureFixture[str], tmp_path: pathlib.Path) -> None:
if parameter in ['-Q', '-C'] and not qreader_available:
if parameter in ['-Q', '-C'] and not cv2_available:
return
# Arrange
@@ -1076,6 +1120,12 @@ url: This is just a text file masquerading as an image file.
assert captured.out == ''
def test_next_valid_qr_mode() -> None:
assert extract_otp_secrets.next_valid_qr_mode(extract_otp_secrets.QRMode.CV2, True) == extract_otp_secrets.QRMode.CV2_WECHAT
assert extract_otp_secrets.next_valid_qr_mode(extract_otp_secrets.QRMode.CV2_WECHAT, True) == extract_otp_secrets.QRMode.ZBAR
assert extract_otp_secrets.next_valid_qr_mode(extract_otp_secrets.QRMode.CV2_WECHAT, False) == extract_otp_secrets.QRMode.CV2
EXPECTED_STDOUT_FROM_EXAMPLE_EXPORT = '''Name: pi@raspberrypi
Secret: 7KSQL2JTUDIS5EF65KLMRQIIGY
Issuer: raspberrypi