commit 69f2f1f635afcf9c6cdda8d7f4bfb0bc61e29638 Author: Moritz Date: Tue Apr 21 09:51:35 2026 +0200 lazyflat: combined alert + apply behind authenticated web UI Three isolated services (alert scraper, apply HTTP worker, web UI+DB) with argon2 auth, signed cookies, CSRF, rate-limited login, kill switch, apply circuit breaker, audit log, and strict CSP. Co-Authored-By: Claude Opus 4.7 (1M context) diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..56be1c7 --- /dev/null +++ b/.env.example @@ -0,0 +1,72 @@ +# ===================================================== +# lazyflat environment configuration +# Copy to .env for local runs; paste into Coolify for deploy. +# ===================================================== + +# -- Auth (web UI) -------------------------------------- +AUTH_USERNAME=moritz +# Generate with: +# python -c "from argon2 import PasswordHasher; print(PasswordHasher().hash('YOUR_PASSWORD'))" +AUTH_PASSWORD_HASH= + +# Long random string — used to sign session & CSRF cookies. +# Generate with: python -c "import secrets; print(secrets.token_urlsafe(48))" +SESSION_SECRET= + +# Must be "true" when running behind HTTPS (i.e. Coolify/Traefik). Set to "false" for plain-http local dev. +COOKIE_SECURE=true +SESSION_MAX_AGE_SECONDS=604800 + +LOGIN_RATE_LIMIT=5 +LOGIN_RATE_WINDOW_SECONDS=900 + +# -- Internal service auth ------------------------------ +# Shared between web <-> apply and alert -> web. Generate with: +# python -c "import secrets; print(secrets.token_urlsafe(48))" +INTERNAL_API_KEY= + +# -- Alert / scraping ----------------------------------- +SLEEP_INTERVALL=60 +GMAPS_API_KEY= +BERLIN_WOHNEN_USERNAME= +BERLIN_WOHNEN_PASSWORD= + +# -- Filter criteria (applied by web) ------------------- +FILTER_ROOMS=2.0,2.5 +FILTER_MAX_RENT=1500 +FILTER_MAX_MORNING_COMMUTE=50 + +# -- Apply (experimental!) ------------------------------ +# Keep SUBMIT_FORMS=False until you have verified the provider flow end-to-end. +SUBMIT_FORMS=False +APPLY_TIMEOUT=600 +APPLY_FAILURE_THRESHOLD=3 +LANGUAGE=de +BROWSER_WIDTH=600 +BROWSER_HEIGHT=800 +BROWSER_LOCALE=de-DE +POST_SUBMISSION_SLEEP_MS=0 + +# Personal info for applications +SALUTATION=Herr +LASTNAME= +FIRSTNAME= +EMAIL= +TELEPHONE= +STREET= +HOUSE_NUMBER= +POSTCODE= +CITY= + +# WBS +IS_POSSESSING_WBS=False +WBS_TYPE=0 +WBS_VALID_TILL=1970-01-01 +WBS_ROOMS=0 +WBS_ADULTS=0 +WBS_CHILDREN=0 +IS_PRIO_WBS=False + +# Optional: immomio login +IMMOMIO_EMAIL= +IMMOMIO_PASSWORD= diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..bbe8f64 --- /dev/null +++ b/.gitignore @@ -0,0 +1,14 @@ +__pycache__/ +*.pyc +*.pyo +.venv/ +venv/ +.env +.env.local +data/ +*.sqlite +*.sqlite-wal +*.sqlite-shm +.DS_Store +.idea/ +.vscode/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..8b04227 --- /dev/null +++ b/README.md @@ -0,0 +1,87 @@ +# lazyflat + +Combined deployment of **flat-alert** (reliable scraper) and **flat-apply** +(experimental auto‑applier) behind a single authenticated web UI. + +## Architecture + +Three isolated containers on one internal Docker network: + +``` +┌─────────┐ POST /internal/flats ┌─────────┐ POST /apply ┌─────────┐ +│ alert │ ────────────────────────► │ web │ ────────────────► │ apply │ +│ scraper │ │ UI+DB │ │ browser │ +└─────────┘ └─────────┘ └─────────┘ + ▲ + │ HTTPS (Coolify / Traefik) + │ + user (auth) +``` + +* **`alert/`** — scrapes inberlinwohnen.de (unchanged logic) and posts each + discovered flat to `web`. Zero external dependencies on `apply`, so `apply` + crashes can never bring the alerter down. +* **`apply/`** — FastAPI wrapper around the experimental Playwright applier. + Only accepts requests with a shared `X-Internal-Api-Key`. Not exposed publicly. +* **`web/`** — FastAPI + Jinja + HTMX dashboard. The only public service. + Owns the SQLite database, auth, and orchestration. + +## Safety / isolation + +Because `apply/` is still experimental, the system is hardened around it: + +| Control | Behavior | +|-----------------------|-----------------------------------------------------------------------| +| Separate container | A crashed `apply` does not take `alert` or `web` with it. | +| Internal-only network | `apply` is not reachable from the internet; requires internal API key.| +| Default mode `manual` | New flats are just shown in the UI; apply runs **only** on click. | +| Circuit breaker | N consecutive apply failures auto-disable further apply calls. | +| Kill switch | One-click button in the UI that blocks all apply activity. | +| `SUBMIT_FORMS=False` | Default for `apply` — runs the full flow **without** final submit. | +| Audit log | Every auth event, mode change, and apply is recorded. | + +## Web security + +* Argon2id password hashes (`argon2-cffi`), constant-time compare. +* Session cookie: signed with `itsdangerous`, `HttpOnly`, `Secure`, `SameSite=Strict`. +* CSRF: synchronizer token bound to the session on every state-changing form. +* Login rate limit (in-memory, per-IP). +* Strict `Content-Security-Policy`, `X-Frame-Options: DENY`, `noindex`. +* `/internal/*` endpoints gated by a shared `INTERNAL_API_KEY` and never exposed by Coolify. + +## Deployment on Coolify + +1. **Create repo**: push this monorepo to `ssh://git@git.moritz.run:2222/moritz/lazyflat.git`. +2. **New Coolify resource** → *Docker Compose* → point it at this repo. Coolify + will read `docker-compose.yml` and deploy all three services on one network. +3. **Domain**: set `flat.lab.moritz.run` on the `web` service only. Coolify + (Traefik) handles TLS. Do **not** set a domain on `alert` or `apply`. +4. **Secrets**: paste the environment variables from `.env.example` into + Coolify's env UI. At minimum you need: + * `AUTH_USERNAME`, `AUTH_PASSWORD_HASH` + * `SESSION_SECRET`, `INTERNAL_API_KEY` + * `GMAPS_API_KEY`, `BERLIN_WOHNEN_USERNAME`, `BERLIN_WOHNEN_PASSWORD` + * personal info + WBS for `apply` +5. **Generate the password hash**: + ```bash + python -c "from argon2 import PasswordHasher; print(PasswordHasher().hash(''))" + ``` +6. **Generate secrets**: + ```bash + python -c "import secrets; print(secrets.token_urlsafe(48))" # SESSION_SECRET + python -c "import secrets; print(secrets.token_urlsafe(48))" # INTERNAL_API_KEY + ``` +7. First apply launch should stay in **manual** mode with + `SUBMIT_FORMS=False` until each provider is verified end-to-end. + +## Local development + +```bash +cp .env.example .env +# fill in AUTH_PASSWORD_HASH, SESSION_SECRET, INTERNAL_API_KEY, creds +docker compose up -d --build +# open http://localhost:8000 (set COOKIE_SECURE=false for plain http!) +``` + +To also expose the web port locally, add `ports: ["8000:8000"]` under `web:` in +`docker-compose.yml` (the Coolify production compose doesn't publish host ports). diff --git a/alert/Dockerfile b/alert/Dockerfile new file mode 100644 index 0000000..7b3f32c --- /dev/null +++ b/alert/Dockerfile @@ -0,0 +1,10 @@ +FROM python:3.11-slim + +WORKDIR /app + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY . . + +CMD ["python", "main.py"] diff --git a/alert/LICENSE b/alert/LICENSE new file mode 100644 index 0000000..f288702 --- /dev/null +++ b/alert/LICENSE @@ -0,0 +1,674 @@ + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. diff --git a/alert/flat.py b/alert/flat.py new file mode 100644 index 0000000..f42ade5 --- /dev/null +++ b/alert/flat.py @@ -0,0 +1,70 @@ +import re +from urllib.parse import quote +from rich.markup import escape + +import maps + + +class Flat: + def __init__(self, data): + self.link = data.get('link', '') + self.address = data.get('Adresse', '') + self.rooms = self._parse_german_float(data.get('Zimmeranzahl', '0')) + self.size = self._parse_german_float(data.get('Wohnfläche', '0')) + self.cold_rent = self._parse_german_float(data.get('Kaltmiete', '0')) + self.utilities = self._parse_german_float(data.get('Nebenkosten', '0')) + self.total_rent = self._parse_german_float(data.get('Gesamtmiete', '0')) + self.available_from = data.get('Bezugsfertig ab', '') + self.published_on = data.get('Eingestellt am', '') + self.wbs = data.get('WBS', '') + self.floor = data.get('Etage', '') + self.bathrooms = data.get('Badezimmer', '') + self.year_built = data.get('Baujahr', '') + self.heating = data.get('Heizung', '') + self.energy_carrier = data.get('Hauptenergieträger', '') + self.energy_value = data.get('Energieverbrauchskennwert', '') + self.energy_certificate = data.get('Energieausweis', '') + self.raw_data = data + self.id = self.link # we could use data.get('id', None) but link is easier to debug + self.gmaps = maps.Maps() + self._connectivity = None + self.address_link_gmaps = f"https://www.google.com/maps/search/?api=1&query={quote(self.address)}" + + def __str__(self): + # URL encode the link to ensure it doesn't contain characters that break markup + # We preserve characters that are standard in URLs but encode problematic ones like brackets and spaces + safe_chars = ":/?#@!$&'()*+,;=%-" + escaped_link = quote(self.link, safe=safe_chars) + return f"[link={escaped_link}]{escape(self.address)}[/link]" + + def _parse_german_float(self, text): + if not text: + return 0.0 + clean_text = re.sub(r'[^\d,.]', '', text) + clean_text = clean_text.replace('.', '').replace(',', '.') + try: + return float(clean_text) + except ValueError: + return 0.0 + + @property + def sqm_price(self): + if self.size > 0: + return self.total_rent / self.size + return 0.0 + + @property + def connectivity(self): + if not self._connectivity: + self._connectivity = self.gmaps.calculate_score(self.address) + return self._connectivity + + @property + def display_address(self): + if ',' in self.address: + parts = self.address.split(',', 1) + street_part = parts[0].strip() + city_part = parts[1].replace(',', '').strip() + return f"{street_part}\n{city_part}" + else: + return self.address diff --git a/alert/main.py b/alert/main.py new file mode 100644 index 0000000..10324e8 --- /dev/null +++ b/alert/main.py @@ -0,0 +1,100 @@ +import logging +import time +from rich.console import Console +from rich.logging import RichHandler + +from flat import Flat +from scraper import Scraper +from settings import TIME_INTERVALL +from utils import hash_any_object +from web_client import WebClient + + +def setup_logging(): + logging.basicConfig( + level=logging.INFO, + format="%(message)s", + datefmt="[%X]", + handlers=[RichHandler(markup=True, console=Console(width=110))], + ) + logging.getLogger("googlemaps").setLevel(logging.WARNING) + logging.getLogger("urllib3").setLevel(logging.WARNING) + + +logger = logging.getLogger("alert") +setup_logging() + + +class FlatAlerter: + def __init__(self): + self.web = WebClient() + self.last_response_hash = "" + + def _flat_payload(self, flat: Flat) -> dict: + c = flat.connectivity + return { + "id": flat.id, + "link": flat.link, + "address": flat.address, + "rooms": flat.rooms, + "size": flat.size, + "cold_rent": flat.cold_rent, + "utilities": flat.utilities, + "total_rent": flat.total_rent, + "sqm_price": flat.sqm_price, + "available_from": flat.available_from, + "published_on": flat.published_on, + "wbs": flat.wbs, + "floor": flat.floor, + "bathrooms": flat.bathrooms, + "year_built": flat.year_built, + "heating": flat.heating, + "energy_carrier": flat.energy_carrier, + "energy_value": flat.energy_value, + "energy_certificate": flat.energy_certificate, + "address_link_gmaps": flat.address_link_gmaps, + "connectivity": { + "morning_time": c.get("morning_time", 0), + "morning_transfers": c.get("morning_transfers", 0), + "night_time": c.get("night_time", 0), + "night_transfers": c.get("night_transfers", 0), + }, + "raw_data": flat.raw_data, + } + + def scan(self): + logger.info("starting scan") + scraper = Scraper() + if not scraper.login(): + return + flats_data = scraper.get_flats() + + response_hashed = hash_any_object(flats_data) + if response_hashed == self.last_response_hash: + logger.info("no change since last scan") + return + self.last_response_hash = response_hashed + + for number, data in enumerate(flats_data, 1): + flat = Flat(data) + logger.info(f"{str(number).rjust(2)}: submitting {flat}") + payload = self._flat_payload(flat) + if not self.web.submit_flat(payload): + logger.warning(f"\tcould not submit {flat.id} to web, will retry next loop") + logger.info("scan finished") + + +if __name__ == "__main__": + logger.info("starting lazyflat alert service") + alerter = FlatAlerter() + while True: + try: + alerter.scan() + alerter.web.heartbeat() + logger.info(f"sleeping for {TIME_INTERVALL} seconds") + time.sleep(TIME_INTERVALL) + except KeyboardInterrupt: + break + except Exception: + logger.exception("unexpected error") + time.sleep(60) diff --git a/alert/maps.py b/alert/maps.py new file mode 100644 index 0000000..b024658 --- /dev/null +++ b/alert/maps.py @@ -0,0 +1,90 @@ +import logging + +import googlemaps +from datetime import datetime, timedelta, time as dt_time +from settings import GMAPS_API_KEY + +logger = logging.getLogger("flat-alert") + +class Maps: + DESTINATIONS = { + "Hbf": "Berlin Hauptbahnhof", + "Friedrichstr": "Friedrichstraße, Berlin", + "Kotti": "Kottbusser Tor, Berlin", + "Warschauer": "Warschauer Straße, Berlin", + "Ostkreuz": "Ostkreuz, Berlin", + "Nollendorf": "Nollendorfplatz, Berlin", + "Zoo": "Zoologischer Garten, Berlin", + "Kudamm": "Kurfürstendamm, Berlin", + "Gesundbrunnen": "Gesundbrunnen, Berlin", + "Hermannplatz": "Hermannplatz, Berlin" + } + + def __init__(self): + self.gmaps = googlemaps.Client(key=GMAPS_API_KEY) + + def _get_next_weekday(self, date, weekday): + days_ahead = weekday - date.weekday() + if days_ahead <= 0: + days_ahead += 7 + return date + timedelta(days_ahead) + + def _calculate_transfers(self, steps): + transit_count = sum(1 for step in steps if step['travel_mode'] == 'TRANSIT') + return max(0, transit_count - 1) + + def calculate_score(self, origin_address): + now = datetime.now() + # Next Monday 8:00 AM + next_monday = self._get_next_weekday(now, 0) + morning_departure = datetime.combine(next_monday.date(), dt_time(8, 0)) + # Next Sunday 2:00 AM + next_sunday = self._get_next_weekday(now, 6) + night_departure = datetime.combine(next_sunday.date(), dt_time(2, 0)) + + total_morning_minutes = 0 + total_morning_transfers = 0 + total_night_minutes = 0 + total_night_transfers = 0 + dest_count = 0 + + for key, dest_address in self.DESTINATIONS.items(): + # Morning: Flat -> Center + routes_morning = self.gmaps.directions( + origin=origin_address, + destination=dest_address, + mode="transit", + departure_time=morning_departure + ) + + # Night: Center -> Flat + routes_night = self.gmaps.directions( + origin=dest_address, + destination=origin_address, + mode="transit", + departure_time=night_departure + ) + + if routes_morning: + leg = routes_morning[0]['legs'][0] + total_morning_minutes += leg['duration']['value'] / 60 + total_morning_transfers += self._calculate_transfers(leg['steps']) + + if routes_night: + leg = routes_night[0]['legs'][0] + total_night_minutes += leg['duration']['value'] / 60 + total_night_transfers += self._calculate_transfers(leg['steps']) + + dest_count += 1 + + avg_m_time = total_morning_minutes / dest_count if dest_count else 0 + avg_m_trans = total_morning_transfers / dest_count if dest_count else 0 + avg_n_time = total_night_minutes / dest_count if dest_count else 0 + avg_n_trans = total_night_transfers / dest_count if dest_count else 0 + + return { + 'morning_time': avg_m_time, + 'morning_transfers': avg_m_trans, + 'night_time': avg_n_time, + 'night_transfers': avg_n_trans + } diff --git a/alert/paths.py b/alert/paths.py new file mode 100644 index 0000000..d12fff4 --- /dev/null +++ b/alert/paths.py @@ -0,0 +1,7 @@ +import os + +DATA_DIR = "data" +ALREADY_NOTIFIED_FILE = "data/already_notified.txt" + +# create dirs if they do not exist yet. +os.makedirs(DATA_DIR, exist_ok=True) \ No newline at end of file diff --git a/alert/requirements.txt b/alert/requirements.txt new file mode 100644 index 0000000..a636d5d --- /dev/null +++ b/alert/requirements.txt @@ -0,0 +1,16 @@ +beautifulsoup4==4.14.3 +bs4==0.0.2 +certifi==2026.1.4 +charset-normalizer==3.4.4 +dotenv==0.9.9 +googlemaps==4.10.0 +idna==3.11 +markdown-it-py==4.0.0 +mdurl==0.1.2 +Pygments==2.19.2 +python-dotenv==1.2.1 +requests==2.32.5 +rich==14.3.1 +soupsieve==2.8.3 +typing_extensions==4.15.0 +urllib3==2.6.3 diff --git a/alert/scraper.py b/alert/scraper.py new file mode 100644 index 0000000..bef85d7 --- /dev/null +++ b/alert/scraper.py @@ -0,0 +1,92 @@ +import requests +import re +import logging +from bs4 import BeautifulSoup +from settings import BERLIN_WOHNEN_USERNAME, BERLIN_WOHNEN_PASSWORD + +logger = logging.getLogger("flat-alert") + +class Scraper: + URL_LOGIN = 'https://www.inberlinwohnen.de/login' + URL_FINDER = 'https://www.inberlinwohnen.de/mein-bereich/wohnungsfinder' + BASE_URL = 'https://www.inberlinwohnen.de' + + HEADERS = { + 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36', + 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8', + 'Accept-Language': 'de,en;q=0.9,en-US;q=0.8', + 'Cache-Control': 'max-age=0', + 'Upgrade-Insecure-Requests': '1', + } + + def __init__(self): + self.session = requests.Session() + self.session.headers.update(self.HEADERS) + + def login(self): + logger.info("fetching inberlinwohnen.de login page") + resp_login_page = self.session.get(self.URL_LOGIN, timeout=30) + token_search = re.search(r'name="csrf-token" content="([^"]+)"', resp_login_page.text) + if not token_search: + logger.critical("no CSRF token found on login page.") + return False + csrf_token = token_search.group(1) + + payload_login = { + '_token': csrf_token, + 'email': BERLIN_WOHNEN_USERNAME, + 'password': BERLIN_WOHNEN_PASSWORD, + 'remember': 'on' + } + headers_login = self.HEADERS.copy() + headers_login['Referer'] = self.URL_LOGIN + + logger.info("attempting login") + resp_post = self.session.post(self.URL_LOGIN, data=payload_login, headers=headers_login, timeout=30) + + if not resp_post.ok or "login" in resp_post.url: + logger.critical("login failed") + logger.info(f"status code: {resp_post.status_code}") + logger.info(f"url: {resp_post.url}") + return False + logger.info("login successful") + return True + + def get_flats(self): + logger.info("fetching flat list") + self.session.headers.update({'Referer': 'https://www.inberlinwohnen.de/mein-bereich'}) + resp_finder = self.session.get(self.URL_FINDER, timeout=30) + + soup = BeautifulSoup(resp_finder.text, 'html.parser') + apartment_divs = soup.find_all('div', id=re.compile(r'^apartment-\d+')) + + logger.info(f"found {len(apartment_divs)} apartments on page.") + + flats_data = [] + for div in apartment_divs: + flat_id = div['id'].replace('apartment-', '') + data = {'id': flat_id} + + link_elem = None + for link in div.find_all('a'): + if "alle details" in link.get_text(strip=True).lower(): + link_elem = link.get('href') + break + + if link_elem: + data['link'] = link_elem if link_elem.startswith('http') else self.BASE_URL + link_elem + else: + data['link'] = self.BASE_URL + + details_list = div.find('dl') + if details_list: + dt_elements = details_list.find_all('dt') + for dt in dt_elements: + key = dt.get_text(strip=True).rstrip(':') + dd = dt.find_next_sibling('dd') + if dd: + data[key] = dd.get_text(strip=True) + + flats_data.append(data) + + return flats_data diff --git a/alert/settings.py b/alert/settings.py new file mode 100644 index 0000000..f6eec2c --- /dev/null +++ b/alert/settings.py @@ -0,0 +1,26 @@ +import sys +from os import getenv +from dotenv import load_dotenv + +load_dotenv() + + +def _required(key: str) -> str: + val = getenv(key) + if not val: + print(f"missing required env var: {key}", file=sys.stderr) + sys.exit(1) + return val + + +LANGUAGE: str = getenv("LANGUAGE", "en") +TIME_INTERVALL: int = int(getenv("SLEEP_INTERVALL", "60")) + +# web backend: alert POSTs discovered flats here +WEB_URL: str = getenv("WEB_URL", "http://web:8000") +INTERNAL_API_KEY: str = _required("INTERNAL_API_KEY") + +# secrets +GMAPS_API_KEY: str = _required("GMAPS_API_KEY") +BERLIN_WOHNEN_USERNAME: str = _required("BERLIN_WOHNEN_USERNAME") +BERLIN_WOHNEN_PASSWORD: str = _required("BERLIN_WOHNEN_PASSWORD") diff --git a/alert/utils.py b/alert/utils.py new file mode 100644 index 0000000..6b4e22a --- /dev/null +++ b/alert/utils.py @@ -0,0 +1,6 @@ +import hashlib +import pickle + +def hash_any_object(var): + byte_stream = pickle.dumps(var) + return hashlib.sha256(byte_stream).hexdigest() \ No newline at end of file diff --git a/alert/web_client.py b/alert/web_client.py new file mode 100644 index 0000000..47fec9f --- /dev/null +++ b/alert/web_client.py @@ -0,0 +1,39 @@ +import logging +import requests + +from settings import WEB_URL, INTERNAL_API_KEY + +logger = logging.getLogger("alert") + + +class WebClient: + def __init__(self, base_url: str = WEB_URL): + self.base_url = base_url.rstrip("/") + self.headers = {"X-Internal-Api-Key": INTERNAL_API_KEY} + + def submit_flat(self, flat_payload: dict) -> bool: + try: + r = requests.post( + f"{self.base_url}/internal/flats", + json=flat_payload, + headers=self.headers, + timeout=10, + ) + if r.status_code >= 400: + logger.error(f"web rejected flat: {r.status_code} {r.text[:300]}") + return False + return True + except requests.RequestException as e: + logger.error(f"web unreachable: {e}") + return False + + def heartbeat(self) -> None: + try: + requests.post( + f"{self.base_url}/internal/heartbeat", + json={"service": "alert"}, + headers=self.headers, + timeout=5, + ) + except requests.RequestException: + pass diff --git a/apply/Dockerfile b/apply/Dockerfile new file mode 100644 index 0000000..c51ec02 --- /dev/null +++ b/apply/Dockerfile @@ -0,0 +1,16 @@ +FROM mcr.microsoft.com/playwright/python:v1.58.0-noble + +ENV PYTHONUNBUFFERED=1 +WORKDIR /app + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY . . + +EXPOSE 8000 + +HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \ + CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=3).status==200 else 1)" + +CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] diff --git a/apply/LICENSE b/apply/LICENSE new file mode 100644 index 0000000..f288702 --- /dev/null +++ b/apply/LICENSE @@ -0,0 +1,674 @@ + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. diff --git a/apply/actions.py b/apply/actions.py new file mode 100644 index 0000000..9f7a2f4 --- /dev/null +++ b/apply/actions.py @@ -0,0 +1,40 @@ +from contextlib import asynccontextmanager +from playwright.async_api import async_playwright, ViewportSize +from reportlab.pdfgen import canvas + +from settings import * +import logging + +logger = logging.getLogger("flat-apply") + +@asynccontextmanager +async def open_page(url): + async with async_playwright() as p: + browser = await p.chromium.launch( + headless=HEADLESS, + args=["--disable-blink-features=AutomationControlled"] + ) + + context = await browser.new_context( + viewport=ViewportSize({ + "width": BROWSER_WIDTH, + "height": BROWSER_HEIGHT}), + locale=BROWSER_LOCALE + ) + + page = await context.new_page() + + await page.goto(url) + await page.wait_for_load_state("networkidle") + + try: + yield page + finally: + await page.wait_for_timeout(POST_SUBMISSION_SLEEP_MS) + await browser.close() + +def create_dummy_pdf(): + logger.info("creating dummy pdf") + c = canvas.Canvas("DummyPDF.pdf") + c.drawString(100, 750, "Hello! This is a dummy PDF file.") + c.save() diff --git a/apply/classes/application_result.py b/apply/classes/application_result.py new file mode 100644 index 0000000..231d48f --- /dev/null +++ b/apply/classes/application_result.py @@ -0,0 +1,33 @@ +from language import _ +import logging + +logger = logging.getLogger("flat-apply") + +class ApplicationResult: + def __init__(self, success: bool, message: str=""): + self.success = success + self.message = message + + def __str__(self): + string = "" + if self.success: + string += _("application_success") + else: + string += _("application_failed") + + if self.message: + string += "\n" + string += self.message + return string + + def __repr__(self): + string = "" + if self.success: + string += "success" + else: + string += "failed" + + if self.message: + string += ": " + string += self.message + return string diff --git a/apply/language.py b/apply/language.py new file mode 100644 index 0000000..2b4e8c3 --- /dev/null +++ b/apply/language.py @@ -0,0 +1,19 @@ +import tomllib + +from settings import * +from paths import * +import logging + +logger = logging.getLogger("flat-apply") + +with open(TRANSLATIONS_FILE, "rb") as f: + TRANSLATIONS = tomllib.load(f) + +def get_text(message): + if message not in TRANSLATIONS.keys(): + raise KeyError(f"{message} is not a valid translation.") + if LANGUAGE not in TRANSLATIONS[message].keys(): + raise KeyError(f"there is no {LANGUAGE} translation for {message}.") + return TRANSLATIONS[message][LANGUAGE] + +_ = get_text diff --git a/apply/main.py b/apply/main.py new file mode 100644 index 0000000..79f19d8 --- /dev/null +++ b/apply/main.py @@ -0,0 +1,83 @@ +import logging +from contextlib import asynccontextmanager +from urllib.parse import urlparse + +from fastapi import Depends, FastAPI, Header, HTTPException, status +from pydantic import BaseModel +from rich.console import Console +from rich.logging import RichHandler + +import providers +from classes.application_result import ApplicationResult +from language import _ +from settings import INTERNAL_API_KEY, log_settings + + +def setup_logging(): + logging.basicConfig( + level=logging.WARNING, + format="%(message)s", + datefmt="[%X]", + handlers=[RichHandler(markup=True, console=Console(width=110))], + ) + logging.getLogger("flat-apply").setLevel(logging.DEBUG) + + +logger = logging.getLogger("flat-apply") +setup_logging() + + +class ApplyRequest(BaseModel): + url: str + + +class ApplyResponse(BaseModel): + success: bool + message: str + + +def require_api_key(x_internal_api_key: str | None = Header(default=None)) -> None: + if not INTERNAL_API_KEY: + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail="apply service has no INTERNAL_API_KEY configured", + ) + if x_internal_api_key != INTERNAL_API_KEY: + raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="invalid api key") + + +@asynccontextmanager +async def lifespan(_app: FastAPI): + log_settings() + logger.info(f"apply ready, providers: {sorted(providers.PROVIDERS)}") + yield + + +app = FastAPI(lifespan=lifespan, title="lazyflat-apply") + + +@app.get("/health") +def health(): + return {"status": "ok"} + + +@app.post("/apply", response_model=ApplyResponse, dependencies=[Depends(require_api_key)]) +async def apply(req: ApplyRequest): + url = req.url.strip() + domain = urlparse(url).netloc.lower().removeprefix("www.") + logger.info(f"apply request for domain={domain} url={url}") + + if domain not in providers.PROVIDERS: + logger.warning(f"unsupported provider: {domain}") + result = ApplicationResult(False, message=_("unsupported_association")) + return ApplyResponse(success=result.success, message=str(result)) + + try: + provider = providers.PROVIDERS[domain] + result = await provider.apply_for_flat(url) + logger.info(f"application result: {repr(result)}") + except Exception as e: + logger.exception("error while applying") + result = ApplicationResult(False, f"Script Error:\n{e}") + + return ApplyResponse(success=result.success, message=str(result)) diff --git a/apply/paths.py b/apply/paths.py new file mode 100644 index 0000000..81e0679 --- /dev/null +++ b/apply/paths.py @@ -0,0 +1,19 @@ +from pathlib import Path +import logging + +logger = logging.getLogger("flat-apply") + +def get_project_root() -> Path: + """Returns the root directory of the project.""" + current_path = Path(__file__).resolve() + + for parent in current_path.parents: + if (parent / ".git").exists(): + return parent + if (parent / "requirements.txt").exists(): + return parent + + return current_path.parent.parent + +ROOT_DIR = get_project_root() +TRANSLATIONS_FILE = ROOT_DIR / "translations.toml" diff --git a/apply/providers/__init__.py b/apply/providers/__init__.py new file mode 100644 index 0000000..fd276a0 --- /dev/null +++ b/apply/providers/__init__.py @@ -0,0 +1,18 @@ +from providers._provider import Provider + +import pkgutil +import importlib + + +# This code runs the moment anyone imports the 'providers' package +__all__ = [] +for loader, module_name, is_pkg in pkgutil.walk_packages(__path__): + full_module_name = f"{__name__}.{module_name}" + importlib.import_module(full_module_name) + __all__.append(module_name) + + +PROVIDERS = dict() +for provider_class in Provider.__subclasses__(): + provider_instance = provider_class() + PROVIDERS[provider_instance.domain] = provider_instance diff --git a/apply/providers/_provider.py b/apply/providers/_provider.py new file mode 100644 index 0000000..cddbcfe --- /dev/null +++ b/apply/providers/_provider.py @@ -0,0 +1,22 @@ +import asyncio +from abc import ABC, abstractmethod +import logging + +from classes.application_result import ApplicationResult + +logger = logging.getLogger("flat-apply") + +class Provider(ABC): + @property + @abstractmethod + def domain(self) -> str: + """every flat provider needs a domain""" + pass + + @abstractmethod + async def apply_for_flat(self, url: str) -> ApplicationResult: + """every flat provider needs to be able to apply for flats""" + pass + + def test_apply(self, url): + print(asyncio.run(self.apply_for_flat(url))) diff --git a/apply/providers/degewo.py b/apply/providers/degewo.py new file mode 100644 index 0000000..7c2ef05 --- /dev/null +++ b/apply/providers/degewo.py @@ -0,0 +1,129 @@ +from actions import * +from language import _ +from classes.application_result import ApplicationResult +from providers._provider import Provider +from settings import * +import logging + +logger = logging.getLogger("flat-apply") + +class Degewo(Provider): + @property + def domain(self) -> str: + return "degewo.de" + + async def apply_for_flat(self, url) -> ApplicationResult: + async with open_page(url) as page: + logger.info("\tSTEP 1: accepting cookies") + cookie_accept_btn = page.locator("#cookie-consent-submit-all") + if await cookie_accept_btn.is_visible(): + await cookie_accept_btn.click() + logger.debug("\t\tcookie accept button clicked") + else: + logger.debug("\t\tno cookie accept button found") + + logger.info("\tSTEP 2: check if the page was not found") + if page.url == "https://www.degewo.de/immosuche/404": + logger.debug("\t\t'page not found' message found - returning") + return ApplicationResult( + success=False, + message=_("ad_offline")) + logger.debug("\t\t'page not found' message not found") + + logger.info("\tSTEP 3: check if the ad is deactivated") + if await page.locator("span", has_text="Inserat deaktiviert").is_visible(): + logger.debug("\t\t'ad deactivated' message found - returning") + return ApplicationResult( + success=False, + message=_("ad_deactivated")) + logger.debug("\t\t'ad deactivated' message not found") + + logger.info("\tSTEP 4: check if the page moved") + if await page.locator("h1", has_text="Diese Seite ist umgezogen!").is_visible(): + logger.debug("\t\t'page moved' message found - returning") + return ApplicationResult( + success=False, + message=_("ad_offline")) + logger.debug("\t\t'page moved' message not found") + + + logger.info("\tSTEP 5: go to the application form") + await page.get_by_role("link", name="Kontakt").click() + + logger.info("\tSTEP 6: find the form iframe") + form_frame = page.frame_locator("iframe[src*='wohnungshelden']") + + logger.info("\tSTEP 7: fill the form") + await form_frame.locator("#salutation").fill(SALUTATION) + await form_frame.get_by_role("option", name=SALUTATION, exact=True).click() + await form_frame.locator("#firstName").fill(FIRSTNAME) + await form_frame.locator("#lastName").fill(LASTNAME) + await form_frame.locator("#email").fill(EMAIL) + await form_frame.locator("input[title='Telefonnummer']").fill(TELEPHONE) + await form_frame.locator("input[title='Anzahl einziehende Personen']").fill(str(PERSON_COUNT)) + await page.wait_for_timeout(1000) + wbs_question = form_frame.locator("input[id*='wbs_available'][id$='Ja']") + if await wbs_question.is_visible(): + if not IS_POSSESSING_WBS: + return ApplicationResult(False, _("wbs_required")) + await wbs_question.click() + await form_frame.locator("input[title='WBS gültig bis']").fill(WBS_VALID_TILL.strftime("%d.%m.%Y")) + await page.wait_for_timeout(1000) + wbs_rooms_select = form_frame.locator("ng-select[id*='wbs_max_number_rooms']") + await wbs_rooms_select.click() + await page.wait_for_timeout(1000) + correct_wbs_room_option = form_frame.get_by_role("option", name=str(WBS_ROOMS), exact=True) + await correct_wbs_room_option.click() + await page.wait_for_timeout(1000) + await form_frame.locator("ng-select[id*='fuer_wen_ist_wohnungsanfrage']").click() + await page.wait_for_timeout(1000) + await form_frame.get_by_role("option", name="Für mich selbst").click() + + logger.info("\tSTEP 8: submit the form") + if not SUBMIT_FORMS: + logger.debug(f"\t\tdry run - not submitting") + return ApplicationResult(success=True, message=_("application_success_dry")) + await form_frame.locator("button[data-cy*='btn-submit']").click() + await page.wait_for_timeout(3000) + + logger.info("\tSTEP 9: check the success") + if await page.locator("h4", has_text="Vielen Dank für das Übermitteln Ihrer Informationen. Sie können dieses Fenster jetzt schließen.").is_visible(): + logger.info(f"\t\tsuccess detected by heading") + return ApplicationResult(success=True) + elif await self.is_missing_fields_warning(page): + logger.warning(f"\t\tmissing fields warning detected") + return ApplicationResult(success=False, message=_("missing_fields")) + elif await self.is_already_applied_warning(page): + logger.warning(f"\t\talready applied warning detected") + return ApplicationResult(success=False, message=_("already_applied")) + logger.warning(f"\t\tsubmit conformation not found") + return ApplicationResult(success=False, message=_("submit_conformation_msg_not_found")) + + async def is_already_applied_warning(self, page): + await page.wait_for_timeout(1000) + form_iframe = page.frame_locator("iframe[src*='wohnungshelden']") + already_applied_warning = form_iframe.locator("span.ant-alert-message", + has_text="Es existiert bereits eine Anfrage mit dieser E-Mail Adresse") + if await already_applied_warning.first.is_visible(): + return True + return False + + async def is_missing_fields_warning(self, page): + await page.wait_for_timeout(1000) + form_iframe = page.frame_locator("iframe[src*='wohnungshelden']") + already_applied_warning = form_iframe.locator("span.ant-alert-message", + has_text="Es wurden nicht alle Felder korrekt befüllt. Bitte prüfen Sie ihre Eingaben") + if await already_applied_warning.first.is_visible(): + return True + return False + +if __name__ == "__main__": + # url = "https://www.degewo.de/immosuche/details/neubau-mit-wbs-140-160-180-220-mit-besonderem-wohnbedarf-1" # already applied + # url = "https://www.degewo.de/immosuche/details/wohnung-sucht-neuen-mieter-1" # angebot geschlossen + # url = "https://www.degewo.de/immosuche/details/wohnung-sucht-neuen-mieter-145" # seite nicht gefunden + # url = "https://www.degewo.de/immosuche/details/1-zimmer-mit-balkon-3" + # url = "https://www.degewo.de/immosuche/details/2-zimmer-in-gropiusstadt-4" + url = "https://www.degewo.de/immosuche/details/2-zimmer-in-gropiusstadt-4" + provider = Degewo() + provider.test_apply(url) + diff --git a/apply/providers/gesobau.py b/apply/providers/gesobau.py new file mode 100644 index 0000000..6c481f0 --- /dev/null +++ b/apply/providers/gesobau.py @@ -0,0 +1,81 @@ +from actions import * +from language import _ +from classes.application_result import ApplicationResult +from providers._provider import Provider +from settings import * +import logging + +logger = logging.getLogger("flat-apply") + +class Gesobau(Provider): + @property + def domain(self) -> str: + return "gesobau.de" + + async def apply_for_flat(self, url) -> ApplicationResult: + async with open_page(url) as page: + logger.info("\tSTEP 1: extracting immomio link") + immomio_link = await page.get_by_role("link", name="Jetzt bewerben").get_attribute("href") + + logger.info("\tSTEP 2: going to auth page") + await page.goto("https://tenant.immomio.com/de/auth/login") + await page.wait_for_timeout(1000) + + logger.info("\tSTEP 3: logging in") + await page.locator('input[name="email"]').fill(IMMOMIO_EMAIL) + await page.get_by_role("button", name="Anmelden").click() + await page.wait_for_timeout(1000) + await page.locator("#password").fill(IMMOMIO_PASSWORD) + await page.locator("#kc-login").click() + await page.wait_for_timeout(1000) + + logger.info("\tSTEP 4: going back to immomio") + await page.goto(immomio_link) + await page.wait_for_timeout(1000) + + logger.info("\tSTEP 5: accepting cookies") + cookie_accept_btn = page.get_by_role("button", name="Alle erlauben") + if await cookie_accept_btn.is_visible(): + await cookie_accept_btn.click() + logger.debug("\t\tcookie accept button clicked") + else: + logger.debug("\t\tno cookie accept button found") + + logger.info("\tSTEP 6: click apply now") + await page.get_by_role("button", name="Jetzt bewerben").click() + await page.wait_for_timeout(3000) + + logger.info("\tSTEP 7: check if already applied") + if page.url == "https://tenant.immomio.com/de/properties/applications": + return ApplicationResult(False, message=_("already_applied")) + + logger.info("\tSTEP 8: clicking answer questions") + answer_questions_btn = page.get_by_role("button", name="Fragen beantworten") + if await answer_questions_btn.is_visible(): + await answer_questions_btn.click() + logger.debug("\t\tanswer questions button clicked") + await page.wait_for_timeout(2000) + else: + logger.debug("\t\tno answer questions button found") + + if await answer_questions_btn.is_visible(): # sometimes this button must be clicked twice + await answer_questions_btn.click() + logger.debug("\t\tanswer questions button clicked") + await page.wait_for_timeout(2000) + + + logger.info("\tSTEP 9: verifying success by answer button vanishing") + if not await answer_questions_btn.is_visible(): # TODO better verify success + logger.info("\t\tsuccess detected by answer button vanishing") + return ApplicationResult(True) + + logger.info("\t\tsubmit conformation not found") + return ApplicationResult(False, _("submit_conformation_msg_not_found")) + + +if __name__ == "__main__": + # url = "https://www.gesobau.de/?immo_ref=10-03239-00007-1185" # already applied + # url = "https://www.gesobau.de/mieten/wohnungssuche/detailseite/florastrasse-10-12179-00002-1002-1d4d1a94-b555-48f8-b06d-d6fc02aecb0d/" + url = "https://www.gesobau.de/mieten/wohnungssuche/detailseite/rolandstrasse-10-03020-00007-1052-7f47d893-e659-4e4f-a7cd-5dcd53f4e6d7/" + provider = Gesobau() + provider.test_apply(url) diff --git a/apply/providers/gewobag.py b/apply/providers/gewobag.py new file mode 100644 index 0000000..73ef640 --- /dev/null +++ b/apply/providers/gewobag.py @@ -0,0 +1,168 @@ +from actions import * +from language import _ +from classes.application_result import ApplicationResult +from providers._provider import Provider +from settings import * +import logging + +logger = logging.getLogger("flat-apply") + +class Gewobag(Provider): + @property + def domain(self) -> str: + return "gewobag.de" + + async def apply_for_flat(self, url) -> ApplicationResult: + async with open_page(url) as page: + logger.info("\tSTEP 1: accepting cookies") + cookie_accept_btn = page.get_by_text("Alle Cookies akzeptieren") + if await cookie_accept_btn.is_visible(): + await cookie_accept_btn.click() + logger.debug("\t\tcookie accept button clicked") + else: + logger.debug("\t\tno cookie accept button found") + + logger.info("\tSTEP 2: check if the page was not found") + if await page.get_by_text("Mietangebot nicht gefunden").first.is_visible(): + logger.debug("\t\t'page not found' message found - returning") + return ApplicationResult( + success=False, + message=_("not_found")) + logger.debug("\t\t'page not found' message not found") + + + logger.info("\tSTEP 3: check if ad is still open") + if await page.locator('#immo-mediation-notice').is_visible(): + logger.debug("\t\tad closed notice found - returning") + return ApplicationResult( + success=False, + message=_("ad_deactivated")) + logger.debug("\t\tno ad closed notice found") + + logger.info("\tSTEP 4: go to the application form") + await page.get_by_role("button", name="Anfrage senden").first.click() + + logger.info("\tSTEP 5: check if the flat is for seniors only") + if await self.is_senior_flat(page): + logger.debug("\t\tflat is for seniors only - returning") + return ApplicationResult(False, _("senior_flat")) + logger.debug("\t\tflat is not seniors only") + + logger.info("\tSTEP 6: check if the flat is for special needs wbs only") + if await self.is_special_needs_wbs(page): + logger.debug("\t\tflat is for special needs wbs only - returning") + return ApplicationResult(False, _("special_need_wbs_flat")) + logger.debug("\t\tflat is not for special needs wbs only") + + logger.info("\tSTEP 7: find the form iframe") + form_iframe = page.frame_locator("#contact-iframe") + + logger.info("\tSTEP 8: define helper functions") + async def fill_field(locator, filling): + logger.debug(f"\t\tfill_field('{locator}', '{filling}')") + field = form_iframe.locator(locator) + if await field.is_visible(): + await field.fill(filling) + await page.wait_for_timeout(100) + else: + logger.debug(f"\t\t\tfield was not found") + + async def select_field(locator, selection): + logger.debug(f"\t\tselect_field('{locator}', '{selection}')") + field = form_iframe.locator(locator) + if await field.is_visible(): + await field.click() + await page.wait_for_timeout(100) + await form_iframe.get_by_role("option", name=selection, exact=True).click() + await page.wait_for_timeout(100) + else: + logger.debug(f"\t\t\tfield was not found") + + async def check_checkbox(locator): + logger.debug(f"\t\tcheck_checkbox('{locator}')") + field = form_iframe.locator(locator) + if await field.first.is_visible(): + await field.evaluate_all("elements => elements.forEach(el => el.click())") + await page.wait_for_timeout(100) + else: + logger.debug(f"\t\t\tfield was not found") + + async def upload_files(locator, files=None): + if not files: + create_dummy_pdf() + files = ["DummyPDF.pdf"] + + logger.debug(f"\t\tupload_files('{locator}', {str(files)})") + wbs_upload_section = form_iframe.locator(locator) + if await wbs_upload_section.count() > 0: + await wbs_upload_section.locator("input[type='file']").set_input_files(files) + await page.wait_for_timeout(2000) + else: + logger.debug(f"\t\t\tfield was not found") + + + logger.info("\tSTEP 9: fill the form") + await select_field("#salutation-dropdown", SALUTATION) + await fill_field("#firstName", FIRSTNAME) + await fill_field("#lastName", LASTNAME) + await fill_field("#email", EMAIL) + await fill_field("#phone-number", TELEPHONE) + await fill_field("#street", STREET) + await fill_field("#house-number", HOUSE_NUMBER) + await fill_field("#zip-code", POSTCODE) + await fill_field("#city", CITY) + await fill_field("input[id*='anzahl_erwachsene']", str(ADULT_COUNT)) + await fill_field("input[id*='anzahl_kinder']", str(CHILDREN_COUNT)) + await fill_field("input[id*='gesamtzahl_der_einziehenden_personen']", str(PERSON_COUNT)) + await check_checkbox("[data-cy*='wbs_available'][data-cy*='-Ja']") + await fill_field("input[id*='wbs_valid_until']", WBS_VALID_TILL.strftime("%d.%m.%Y")) + await select_field("input[id*='wbs_max_number_rooms']", f"{WBS_ROOMS} Räume") + await select_field("input[id*='art_bezeichnung_des_wbs']", f"WBS {WBS_TYPE}") + await select_field("input[id*='fuer_wen']", "Für mich selbst") + await fill_field("input[id*='telephone_number']", TELEPHONE) + await check_checkbox("input[id*='datenschutzhinweis']") + await upload_files("el-application-form-document-upload", files=None) + + logger.info("\tSTEP 10: submit the form") + if not SUBMIT_FORMS: + logger.debug(f"\t\tdry run - not submitting") + return ApplicationResult(True, _("application_success_dry")) + await form_iframe.get_by_role("button", name="Anfrage versenden").click() + await page.wait_for_timeout(5000) + + logger.info("\tSTEP 11: check the success") + if page.url.startswith("https://www.gewobag.de/daten-uebermittelt/"): + logger.info(f"\t\tsuccess detected by page url") + return ApplicationResult(True) + elif self.is_missing_fields_warning(page): + logger.warning(f"\t\tmissing fields warning detected") + return ApplicationResult(False, _("missing_fields")) + else: + logger.warning(f"\t\tneither missing fields nor success detected") + return ApplicationResult(False, _("submit_conformation_msg_not_found")) + + async def is_senior_flat(self, page): + form_iframe = page.frame_locator("#contact-iframe") + return await form_iframe.locator("label[for*='mindestalter_seniorenwohnhaus_erreicht']").first.is_visible() + + async def is_special_needs_wbs(self, page): + form_iframe = page.frame_locator("#contact-iframe") + return await form_iframe.locator("label[for*='wbs_mit_besonderem_wohnbedarf_vorhanden']").first.is_visible() + + async def is_missing_fields_warning(self, page): + form_iframe = page.frame_locator("#contact-iframe") + missing_field_msg = form_iframe.locator("span.ant-alert-message", + has_text="Es wurden nicht alle Felder korrekt befüllt.") + if await missing_field_msg.first.is_visible(): + return True + return False + +if __name__ == "__main__": + #url = "https://www.gewobag.de/fuer-mietinteressentinnen/mietangebote/0100-01036-0601-0286-vms1/" # wbs + #url = "https://www.gewobag.de/fuer-mietinteressentinnen/mietangebote/7100-72401-0101-0011/" # senior + url = "https://www.gewobag.de/fuer-mietinteressentinnen/mietangebote/6011-31046-0105-0045/" # more wbs fields + #url = "https://www.gewobag.de/fuer-mietinteressentinnen/mietangebote/0100-01036-0401-0191/" # special need wbs + #url = "https://www.gewobag.de/fuer-mietinteressentinnen/mietangebote/0100-02571-0103-0169/" + # url = "https://www.gewobag.de/fuer-mietinteressentinnen/mietangebote/0100-02571-0103-169/" # page not found + provider = Gewobag() + provider.test_apply(url) diff --git a/apply/providers/howoge.py b/apply/providers/howoge.py new file mode 100644 index 0000000..f3cc555 --- /dev/null +++ b/apply/providers/howoge.py @@ -0,0 +1,62 @@ +from actions import * +from language import _ +from classes.application_result import ApplicationResult +from providers._provider import Provider +from settings import * +import logging + +logger = logging.getLogger("flat-apply") + +class Howoge(Provider): + @property + def domain(self) -> str: + return "howoge.de" + + async def apply_for_flat(self, url) -> ApplicationResult: + async with open_page(url) as page: + logger.info("\tSTEP 1: accepting cookies") + cookie_accept_btn = page.get_by_role("button", name="Alles akzeptieren") + if await cookie_accept_btn.is_visible(): + await cookie_accept_btn.click() + logger.debug("\t\tcookie accept button clicked") + else: + logger.debug("\t\tno cookie accept button found") + + logger.info("\tSTEP 2: check if the page was not found") + if page.url == "https://www.howoge.de/404": + logger.debug("\t\t'page not found' url found - returning") + return ApplicationResult( + success=False, + message=_("not_found")) + logger.debug("\t\t'page not found' url not found") + + logger.info("\tSTEP 3: go to the application form") + await page.get_by_role("link", name="Besichtigung anfragen").click() + + logger.info("\tSTEP 4: fill the form") + await page.get_by_text("Ja, ich habe die Hinweise zum WBS zur Kenntnis genommen.").click() + await page.get_by_role("button", name="Weiter").click() + await page.get_by_text("Ja, ich habe den Hinweis zum Haushaltsnettoeinkommen zur Kenntnis genommen.").click() + await page.get_by_role("button", name="Weiter").click() + await page.get_by_text("Ja, ich habe den Hinweis zur Bonitätsauskunft zur Kenntnis genommen.").click() + await page.get_by_role("button", name="Weiter").click() + await page.locator("#immo-form-firstname").fill(FIRSTNAME) + await page.locator("#immo-form-lastname").fill(LASTNAME) + await page.locator("#immo-form-email").fill(EMAIL) + + logger.info("\tSTEP 5: submit the form") + if not SUBMIT_FORMS: + logger.debug(f"\t\tdry run - not submitting") + return ApplicationResult(True, _("application_success_dry")) + await page.get_by_role("button", name="Anfrage senden").click() + + logger.info("\tSTEP 6: check the success") + if await page.get_by_role("heading", name="Vielen Dank.").is_visible(): + return ApplicationResult(True) + return ApplicationResult(False, _("submit_conformation_msg_not_found")) + +if __name__ == "__main__": + # url = "https://www.howoge.de/wohnungen-gewerbe/wohnungssuche/detail/1770-26279-6.html" # not found + url = "https://www.howoge.de/immobiliensuche/wohnungssuche/detail/1770-27695-194.html" + provider = Howoge() + provider.test_apply(url) diff --git a/apply/providers/stadtundland.py b/apply/providers/stadtundland.py new file mode 100644 index 0000000..d891ea3 --- /dev/null +++ b/apply/providers/stadtundland.py @@ -0,0 +1,65 @@ +from actions import * +from language import _ +from classes.application_result import ApplicationResult +from providers._provider import Provider +from settings import * +import logging + +logger = logging.getLogger("flat-apply") + +class Stadtundland(Provider): + @property + def domain(self) -> str: + return "stadtundland.de" + + async def apply_for_flat(self, url) -> ApplicationResult: + async with open_page(url) as page: + logger.info("\tSTEP 1: accepting cookies") + cookie_accept_btn = page.get_by_text("Alle akzeptieren") + if await cookie_accept_btn.is_visible(): + await cookie_accept_btn.click() + logger.debug("\t\tcookie accept button clicked") + else: + logger.debug("\t\tno cookie accept button found") + + logger.info("\tSTEP 2: check if ad is still open") + if await page.get_by_role("heading", name="Hier ist etwas schief gelaufen").is_visible(): + logger.debug("\t\tsomething went wrong notice found - returning") + return ApplicationResult( + success=False, + message=_("ad_offline")) + logger.debug("\t\tsomething went wrong notice not found") + + logger.info("\tSTEP 3: fill the form") + await page.locator("#name").fill(FIRSTNAME) + await page.locator("#surname").fill(LASTNAME) + await page.locator("#street").fill(STREET) + await page.locator("#houseNo").fill(HOUSE_NUMBER) + await page.locator("#postalCode").fill(POSTCODE) + await page.locator("#city").fill(CITY) + await page.locator("#phone").fill(TELEPHONE) + await page.locator("#email").fill(EMAIL) + await page.locator("#privacy").check() + await page.locator("#provision").check() + + logger.info("\tSTEP 4: submit the form") + if not SUBMIT_FORMS: + logger.debug(f"\t\tdry run - not submitting") + return ApplicationResult(False, _("application_success_dry")) + await page.get_by_role("button", name="Eingaben prüfen").click() + await page.get_by_role("button", name="Absenden").click() + await page.wait_for_timeout(2000) + + logger.info("\tSTEP 5: check the success") + if await page.locator("p").filter(has_text="Vielen Dank!").is_visible(): + logger.info(f"\t\tsuccess detected by paragraph text") + return ApplicationResult(True) + logger.warning(f"\t\tsuccess message not found") + return ApplicationResult(success=False, message=_("submit_conformation_msg_not_found")) + + +if __name__ == "__main__": + # url = "https://stadtundland.de/wohnungssuche/1001%2F0203%2F00310" # offline + url = "https://stadtundland.de/wohnungssuche/1050%2F8222%2F00091" # wbs + provider = Stadtundland() + provider.test_apply(url) diff --git a/apply/providers/wbm.py b/apply/providers/wbm.py new file mode 100644 index 0000000..51bf0a4 --- /dev/null +++ b/apply/providers/wbm.py @@ -0,0 +1,100 @@ +from actions import * +from language import _ +from classes.application_result import ApplicationResult +from providers._provider import Provider +from settings import * +import logging + +logger = logging.getLogger("flat-apply") + + +class Wbm(Provider): + @property + def domain(self) -> str: + return "wbm.de" + + async def apply_for_flat(self, url) -> ApplicationResult: + async with open_page(url) as page: + logger.info("\tSTEP 1: checking if page not found") + if await page.get_by_role("heading", name="Page Not Found").is_visible(): + logger.debug("\t\t'page not found' message found - returning") + return ApplicationResult( + success=False, + message=_("not_found")) + logger.debug("\t\t'page not found' message not found") + + logger.info("\tSTEP 2: accepting cookies") + cookie_accept_btn = page.get_by_text("Alle zulassen") + if await cookie_accept_btn.is_visible(): + await cookie_accept_btn.click() + logger.debug("\t\tcookie accept button clicked") + else: + logger.debug("\t\tno cookie accept button found") + + logger.info("\tSTEP 3: removing chatbot help icon") + await page.locator('#removeConvaiseChat').click() + + logger.info("\tSTEP 4: checking if ad is offline") + if page.url == "https://www.wbm.de/wohnungen-berlin/angebote/": + logger.debug("\t\t'page not found' url found - returning") + return ApplicationResult( + success=False, + message=_("ad_offline")) + logger.debug("\t\t'page not found' url not found") + + + logger.info("\tSTEP 5: go to the application form") + await page.locator('.openimmo-detail__contact-box-button').click() + + logger.info("\tSTEP 6: filling the application form") + if IS_POSSESSING_WBS: + await page.locator('label[for="powermail_field_wbsvorhanden_1"]').click() + await page.locator("input[name*='[wbsgueltigbis]']").fill(WBS_VALID_TILL.strftime("%Y-%m-%d")) + await page.locator("select[name*='[wbszimmeranzahl]']").select_option(str(WBS_ROOMS)) + await page.locator("#powermail_field_einkommensgrenzenacheinkommensbescheinigung9").select_option(WBS_TYPE) + if IS_PRIO_WBS: + await page.locator("#powermail_field_wbsmitbesonderemwohnbedarf_1").check(force=True) + else: + await page.locator('label[for="powermail_field_wbsvorhanden_2"]').click() + + await page.locator("#powermail_field_anrede").select_option(SALUTATION) + await page.locator("#powermail_field_name").fill(LASTNAME) + await page.locator("#powermail_field_vorname").fill(FIRSTNAME) + await page.locator("#powermail_field_strasse").fill(STREET) + await page.locator("#powermail_field_plz").fill(POSTCODE) + await page.locator("#powermail_field_ort").fill(CITY) + await page.locator("#powermail_field_e_mail").fill(EMAIL) + await page.locator("#powermail_field_telefon").fill(TELEPHONE) + await page.locator("#powermail_field_datenschutzhinweis_1").check(force=True) + + logger.info("\tSTEP 7: submit the form") + if not SUBMIT_FORMS: + logger.debug(f"\t\tdry run - not submitting") + return ApplicationResult(success=True, message=_("application_success_dry")) + await page.get_by_role("button", name="Anfrage absenden").click() + + logger.info("\tSTEP 8: check the success") + if await page.get_by_text("Wir haben Ihre Anfrage für das Wohnungsangebot erhalten.").is_visible(): + logger.info(f"\t\tsuccess detected by text") + return ApplicationResult(True) + elif await self.is_missing_fields_warning(page): + logger.warning(f"\t\tmissing fields warning detected") + return ApplicationResult(False, _("missing_fields")) + else: + logger.warning(f"\t\tneither missing fields nor success detected") + return ApplicationResult(success=False, message=_("submit_conformation_msg_not_found")) + + + async def is_missing_fields_warning(self, page): + missing_field_msg = page.get_by_text("Dieses Feld muss ausgefüllt werden!").first + if await missing_field_msg.first.is_visible(): + return True + return False + + + +if __name__ == "__main__": + # url = "https://www.wbm.de/wohnungen-berlin/angebote/details/4-zimmer-wohnung-in-spandau-1/" # not found + url = "https://www.wbm.de/wohnungen-berlin/angebote/details/wbs-160-180-220-perfekt-fuer-kleine-familien-3-zimmer-wohnung-mit-balkon/" + provider = Wbm() + provider.test_apply(url) diff --git a/apply/requirements.txt b/apply/requirements.txt new file mode 100644 index 0000000..405cfa6 --- /dev/null +++ b/apply/requirements.txt @@ -0,0 +1,26 @@ +CacheControl==0.14.4 +certifi==2026.1.4 +charset-normalizer==3.4.4 +dotenv==0.9.9 +greenlet==3.3.1 +idna==3.11 +lxml==6.0.2 +lxml_html_clean==0.4.3 +markdown-it-py==4.0.0 +mdurl==0.1.2 +msgpack==1.1.2 +fastapi==0.115.5 +pillow==12.1.0 +playwright==1.58.0 +pyee==13.0.0 +uvicorn[standard]==0.32.1 +Pygments==2.19.2 +python-dotenv==1.2.1 +pywebcopy==7.1 +reportlab==4.4.9 +requests==2.32.5 +rich==14.3.1 +setuptools==80.10.2 +six==1.17.0 +typing_extensions==4.15.0 +urllib3==2.6.3 diff --git a/apply/settings.py b/apply/settings.py new file mode 100644 index 0000000..8c30363 --- /dev/null +++ b/apply/settings.py @@ -0,0 +1,110 @@ +import logging +import sys +from datetime import datetime as dt +from os import getenv +from dotenv import load_dotenv + +logger = logging.getLogger("flat-apply") + +load_dotenv() + +def get_env_or_fail(key: str, default: str = None, required: bool = True) -> str: + value = getenv(key, default) + if required and value is None: + logger.error(f"Missing required environment variable: {key}") + sys.exit(1) + return value + +def get_bool_env(key: str, default: str = "False", required: bool = True) -> bool: + return get_env_or_fail(key, default, required).lower() in ("true", "1", "yes", "on") + +def get_int_env(key: str, default: str = None, required: bool = True) -> int: + value_str = get_env_or_fail(key, default, required) + try: + return int(value_str) + except ValueError: + logger.error(f"Environment variable {key} must be an integer. Got: {value_str}") + sys.exit(1) + +def get_date_env(key: str, fmt: str = "%Y-%m-%d", default: str = None, required: bool = True) -> dt: + value_str = get_env_or_fail(key, default, required) + try: + return dt.strptime(value_str, fmt) + except ValueError: + logger.error(f"Environment variable {key} must be a date in format {fmt}. Got: {value_str}") + sys.exit(1) + +# --- General Settings --- +LANGUAGE: str = get_env_or_fail("LANGUAGE", "de", False) + +# --- Browser Settings --- +HEADLESS: bool = get_bool_env("HEADLESS", "True", True) +BROWSER_WIDTH: int = get_int_env("BROWSER_WIDTH", "600", False) +BROWSER_HEIGHT: int = get_int_env("BROWSER_HEIGHT", "800", False) +BROWSER_LOCALE: str = get_env_or_fail("BROWSER_LOCALE", "de-DE", False) +POST_SUBMISSION_SLEEP_MS: int = get_int_env("POST_SUBMISSION_SLEEP_MS", "0", False) + +# --- Automation Mode --- +SUBMIT_FORMS: bool = get_bool_env("SUBMIT_FORMS", "False") + +# --- HTTP Server --- +HTTP_HOST: str = get_env_or_fail("HTTP_HOST", "0.0.0.0", False) +HTTP_PORT: int = get_int_env("HTTP_PORT", "8000", False) + +# --- Personal Information --- +SALUTATION: str = get_env_or_fail("SALUTATION") +LASTNAME: str = get_env_or_fail("LASTNAME") +FIRSTNAME: str = get_env_or_fail("FIRSTNAME") +EMAIL: str = get_env_or_fail("EMAIL") +TELEPHONE: str = get_env_or_fail("TELEPHONE") +STREET: str = get_env_or_fail("STREET") +HOUSE_NUMBER: str = get_env_or_fail("HOUSE_NUMBER") +POSTCODE: str = get_env_or_fail("POSTCODE") +CITY: str = get_env_or_fail("CITY") + +# --- WBS Information --- +IS_POSSESSING_WBS: bool = get_bool_env("IS_POSSESSING_WBS", "False") +WBS_TYPE: str = get_env_or_fail("WBS_TYPE", "0", False) +WBS_VALID_TILL: dt = get_date_env("WBS_VALID_TILL", default="1970-01-01", required=False) +WBS_ROOMS: int = get_int_env("WBS_ROOMS", "0", False) +ADULT_COUNT: int = get_int_env("WBS_ADULTS", "0", False) +CHILDREN_COUNT: int = get_int_env("WBS_CHILDREN", "0", False) +PERSON_COUNT: int = ADULT_COUNT + CHILDREN_COUNT +IS_PRIO_WBS: bool = get_bool_env("IS_PRIO_WBS", "False") + +# --- Secrets --- +IMMOMIO_EMAIL: str = get_env_or_fail("IMMOMIO_EMAIL", required=False) +IMMOMIO_PASSWORD: str = get_env_or_fail("IMMOMIO_PASSWORD", required=False) +INTERNAL_API_KEY: str = get_env_or_fail("INTERNAL_API_KEY") + +def log_settings(): + logger.debug("--- Settings ---") + logger.debug(f"LANGUAGE: {LANGUAGE}") + logger.debug(f"BROWSER_WIDTH: {BROWSER_WIDTH}") + logger.debug(f"BROWSER_HEIGHT: {BROWSER_HEIGHT}") + logger.debug(f"BROWSER_LOCALE: {BROWSER_LOCALE}") + logger.debug(f"POST_SUBMISSION_SLEEP_MS: {POST_SUBMISSION_SLEEP_MS}") + logger.debug(f"HEADLESS: {HEADLESS}") + logger.debug(f"SUBMIT_FORMS: {SUBMIT_FORMS}") + logger.debug(f"HTTP_HOST: {HTTP_HOST}") + logger.debug(f"HTTP_PORT: {HTTP_PORT}") + logger.debug(f"SALUTATION: {SALUTATION}") + logger.debug(f"LASTNAME: {LASTNAME}") + logger.debug(f"FIRSTNAME: {FIRSTNAME}") + logger.debug(f"EMAIL: {EMAIL}") + logger.debug(f"TELEPHONE: {TELEPHONE}") + logger.debug(f"STREET: {STREET}") + logger.debug(f"HOUSE_NUMBER: {HOUSE_NUMBER}") + logger.debug(f"POSTCODE: {POSTCODE}") + logger.debug(f"CITY: {CITY}") + logger.debug(f"IS_POSSESSING_WBS: {IS_POSSESSING_WBS}") + logger.debug(f"WBS_TYPE: {WBS_TYPE}") + logger.debug(f"WBS_VALID_TILL: {WBS_VALID_TILL}") + logger.debug(f"WBS_ROOMS: {WBS_ROOMS}") + logger.debug(f"WBS_ADULTS: {ADULT_COUNT}") + logger.debug(f"WBS_CHILDREN: {CHILDREN_COUNT}") + logger.debug(f"PERSON_COUNT: {PERSON_COUNT}") + logger.debug(f"IS_PRIO_WBS: {IS_PRIO_WBS}") + logger.debug(f"IMMOMIO_EMAIL: {IMMOMIO_EMAIL}") + masked_password = "***" if IMMOMIO_PASSWORD else "None" + logger.debug(f"IMMOMIO_PASSWORD: {masked_password}") diff --git a/apply/translations.toml b/apply/translations.toml new file mode 100644 index 0000000..1281151 --- /dev/null +++ b/apply/translations.toml @@ -0,0 +1,55 @@ +[application_success] +en = "Successfully applied." +de = "Erfolgreich beworben." + +[application_failed] +en = "Unsuccessful application." +de = "Nicht erfolgreich beworben." + +[application_success_dry] +en = "Dryrun. No form submitted." +de = "Trockenlauf. Formular nicht abgeschickt." + +[submit_conformation_msg_not_found] +en = "Submission conformation message not found." +de = "Bestätigungsnachricht nicht gefunden." + +[ad_offline] +en = "Ad offline." +de = "Inserat offline." + +[ad_deactivated] +en = "Ad deactivated." +de = "Inserat deaktiviert." + +[unsupported_association] +en = "This domain is not a supported housing association." +de = "Diese Domain ist keine unterstützte Wohnungsbaugesellschaft" + +[todo_association] +en = "Support for this housing association is not implemented yet." +de = "Diese Wohnungsbaugesellschaft wurde noch nicht implementiert." + +[senior_flat] +en = "This flat is only for elderly people." +de = "Dies ist eine Seniorenwohnung." + +[special_need_wbs_flat] +en = "This flat is only for people with a special needs WBS." +de = "Diese Wohnung ist nur für Menschen mit einem besonderen Wohnbedarf." + +[missing_fields] +en = "Required fields have not been filled." +de = "Pflichtfelder wurden nicht ausgefüllt." + +[not_found] +en = "Page was not found." +de = "Diese Seite wurde nicht gefunden." + +[already_applied] +en = "You already applied for this ad." +de = "Sie haben sich für diese Anzeige bereits beworben." + +[wbs_required] +en = "You need a WBS for this flat." +de = "Sie benötigen einen WBS für diese Wohnung." \ No newline at end of file diff --git a/apply/utils.py b/apply/utils.py new file mode 100644 index 0000000..9824937 --- /dev/null +++ b/apply/utils.py @@ -0,0 +1,12 @@ +import logging + +logger = logging.getLogger("flat-apply") + +def str_to_preview(string, max_length): + if not max_length > 3: + raise ValueError('max_length must be greater than 3') + + first_line = string.split('\n')[0] + if len(first_line) > max_length: + return first_line[:max_length-3] + '...' + return first_line diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..45fff2a --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,82 @@ +services: + web: + build: ./web + container_name: lazyflat-web + restart: unless-stopped + depends_on: + apply: + condition: service_started + environment: + - AUTH_USERNAME=${AUTH_USERNAME} + - AUTH_PASSWORD_HASH=${AUTH_PASSWORD_HASH} + - SESSION_SECRET=${SESSION_SECRET} + - COOKIE_SECURE=${COOKIE_SECURE:-true} + - INTERNAL_API_KEY=${INTERNAL_API_KEY} + - APPLY_URL=http://apply:8000 + - APPLY_TIMEOUT=${APPLY_TIMEOUT:-600} + - APPLY_FAILURE_THRESHOLD=${APPLY_FAILURE_THRESHOLD:-3} + - DATA_DIR=/data + - SESSION_MAX_AGE_SECONDS=${SESSION_MAX_AGE_SECONDS:-604800} + - LOGIN_RATE_LIMIT=${LOGIN_RATE_LIMIT:-5} + - LOGIN_RATE_WINDOW_SECONDS=${LOGIN_RATE_WINDOW_SECONDS:-900} + - FILTER_ROOMS=${FILTER_ROOMS:-2.0,2.5} + - FILTER_MAX_RENT=${FILTER_MAX_RENT:-1500} + - FILTER_MAX_MORNING_COMMUTE=${FILTER_MAX_MORNING_COMMUTE:-50} + volumes: + - lazyflat_data:/data + # Coolify assigns the public port/domain via labels — no host port needed. + expose: + - "8000" + + apply: + build: ./apply + container_name: lazyflat-apply + restart: unless-stopped + # Intentionally NOT exposed to the internet. Reachable only on the compose network. + expose: + - "8000" + environment: + - INTERNAL_API_KEY=${INTERNAL_API_KEY} + - HEADLESS=true + - SUBMIT_FORMS=${SUBMIT_FORMS:-False} + - LANGUAGE=${LANGUAGE:-de} + - BROWSER_WIDTH=${BROWSER_WIDTH:-600} + - BROWSER_HEIGHT=${BROWSER_HEIGHT:-800} + - BROWSER_LOCALE=${BROWSER_LOCALE:-de-DE} + - POST_SUBMISSION_SLEEP_MS=${POST_SUBMISSION_SLEEP_MS:-0} + - SALUTATION=${SALUTATION} + - LASTNAME=${LASTNAME} + - FIRSTNAME=${FIRSTNAME} + - EMAIL=${EMAIL} + - TELEPHONE=${TELEPHONE} + - STREET=${STREET} + - HOUSE_NUMBER=${HOUSE_NUMBER} + - POSTCODE=${POSTCODE} + - CITY=${CITY} + - IS_POSSESSING_WBS=${IS_POSSESSING_WBS:-False} + - WBS_TYPE=${WBS_TYPE:-0} + - WBS_VALID_TILL=${WBS_VALID_TILL:-1970-01-01} + - WBS_ROOMS=${WBS_ROOMS:-0} + - WBS_ADULTS=${WBS_ADULTS:-0} + - WBS_CHILDREN=${WBS_CHILDREN:-0} + - IS_PRIO_WBS=${IS_PRIO_WBS:-False} + - IMMOMIO_EMAIL=${IMMOMIO_EMAIL:-} + - IMMOMIO_PASSWORD=${IMMOMIO_PASSWORD:-} + + alert: + build: ./alert + container_name: lazyflat-alert + restart: unless-stopped + depends_on: + web: + condition: service_started + environment: + - WEB_URL=http://web:8000 + - INTERNAL_API_KEY=${INTERNAL_API_KEY} + - SLEEP_INTERVALL=${SLEEP_INTERVALL:-60} + - GMAPS_API_KEY=${GMAPS_API_KEY} + - BERLIN_WOHNEN_USERNAME=${BERLIN_WOHNEN_USERNAME} + - BERLIN_WOHNEN_PASSWORD=${BERLIN_WOHNEN_PASSWORD} + +volumes: + lazyflat_data: diff --git a/web/Dockerfile b/web/Dockerfile new file mode 100644 index 0000000..ac31eb5 --- /dev/null +++ b/web/Dockerfile @@ -0,0 +1,16 @@ +FROM python:3.12-slim + +ENV PYTHONUNBUFFERED=1 +WORKDIR /app + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY . . + +EXPOSE 8000 + +HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ + CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=3).status==200 else 1)" + +CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000", "--proxy-headers", "--forwarded-allow-ips", "*"] diff --git a/web/app.py b/web/app.py new file mode 100644 index 0000000..e4d9309 --- /dev/null +++ b/web/app.py @@ -0,0 +1,341 @@ +import asyncio +import hmac +import logging +import threading +from contextlib import asynccontextmanager +from typing import Optional + +from fastapi import Depends, FastAPI, Form, HTTPException, Header, Request, Response, status +from fastapi.responses import HTMLResponse, RedirectResponse +from fastapi.staticfiles import StaticFiles +from fastapi.templating import Jinja2Templates + +import db +from apply_client import ApplyClient +from auth import ( + clear_session_cookie, + current_user, + issue_csrf_token, + issue_session_cookie, + rate_limit_login, + require_csrf, + require_user, + verify_password, +) +from settings import ( + APPLY_FAILURE_THRESHOLD, + AUTH_USERNAME, + FILTER_MAX_MORNING_COMMUTE, + FILTER_MAX_RENT, + FILTER_ROOMS, + INTERNAL_API_KEY, +) + + +logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s") +logger = logging.getLogger("web") + +apply_client = ApplyClient() +_apply_lock = threading.Lock() + + +@asynccontextmanager +async def lifespan(_app: FastAPI): + db.init_db() + logger.info("web service started") + yield + + +app = FastAPI(lifespan=lifespan, title="lazyflat", docs_url=None, redoc_url=None, openapi_url=None) +app.mount("/static", StaticFiles(directory="static"), name="static") +templates = Jinja2Templates(directory="templates") + + +# ----------------------------------------------------------------------------- +# Security headers middleware +# ----------------------------------------------------------------------------- +@app.middleware("http") +async def security_headers(request: Request, call_next): + response: Response = await call_next(request) + response.headers.setdefault("X-Frame-Options", "DENY") + response.headers.setdefault("X-Content-Type-Options", "nosniff") + response.headers.setdefault("Referrer-Policy", "strict-origin-when-cross-origin") + response.headers.setdefault("Permissions-Policy", "geolocation=(), camera=(), microphone=()") + response.headers.setdefault( + "Content-Security-Policy", + "default-src 'self'; " + "script-src 'self' https://cdn.tailwindcss.com https://unpkg.com; " + "style-src 'self' https://cdn.tailwindcss.com 'unsafe-inline'; " + "img-src 'self' data:; " + "connect-src 'self'; " + "frame-ancestors 'none';" + ) + return response + + +# ----------------------------------------------------------------------------- +# Helpers +# ----------------------------------------------------------------------------- + +def client_ip(request: Request) -> str: + xff = request.headers.get("x-forwarded-for") + if xff: + return xff.split(",")[0].strip() + return request.client.host if request.client else "unknown" + + +def require_internal(x_internal_api_key: str | None = Header(default=None)) -> None: + if not x_internal_api_key or not hmac.compare_digest(x_internal_api_key, INTERNAL_API_KEY): + raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="invalid internal key") + + +def matches_criteria(payload: dict) -> bool: + rooms = payload.get("rooms") or 0.0 + rent = payload.get("total_rent") or 0.0 + commute = (payload.get("connectivity") or {}).get("morning_time") or 0.0 + if FILTER_ROOMS and rooms not in FILTER_ROOMS: + return False + if rent > FILTER_MAX_RENT: + return False + if commute > FILTER_MAX_MORNING_COMMUTE: + return False + return True + + +def apply_allowed() -> tuple[bool, str]: + if db.get_state("kill_switch") == "1": + return False, "kill switch aktiv" + if db.get_state("apply_circuit_open") == "1": + return False, "circuit breaker offen (zu viele Fehler)" + return True, "" + + +def run_apply(flat_id: str, url: str, triggered_by: str) -> None: + """Run synchronously inside a worker thread via asyncio.to_thread.""" + app_id = db.start_application(flat_id, url, triggered_by) + try: + result = apply_client.apply(url) + except Exception as e: + result = {"success": False, "message": f"client error: {e}"} + + success = bool(result.get("success")) + db.finish_application(app_id, success, result.get("message", "")) + + # Circuit breaker accounting + if success: + db.set_state("apply_recent_failures", "0") + else: + fails = int(db.get_state("apply_recent_failures") or "0") + 1 + db.set_state("apply_recent_failures", str(fails)) + if fails >= APPLY_FAILURE_THRESHOLD: + db.set_state("apply_circuit_open", "1") + db.log_audit("system", "circuit_open", f"{fails} consecutive failures") + + +# ----------------------------------------------------------------------------- +# Public routes +# ----------------------------------------------------------------------------- + +@app.get("/health") +def health(): + return {"status": "ok"} + + +@app.get("/login", response_class=HTMLResponse) +def login_form(request: Request, error: str | None = None): + if current_user(request): + return RedirectResponse("/", status_code=303) + return templates.TemplateResponse("login.html", {"request": request, "error": error}) + + +@app.post("/login") +def login_submit(request: Request, username: str = Form(...), password: str = Form(...)): + ip = client_ip(request) + if not rate_limit_login(ip): + db.log_audit(username or "?", "login_rate_limited", ip=ip) + return templates.TemplateResponse( + "login.html", + {"request": request, "error": "Zu viele Versuche. Bitte später erneut versuchen."}, + status_code=status.HTTP_429_TOO_MANY_REQUESTS, + ) + + if not verify_password(username, password): + db.log_audit(username or "?", "login_failed", ip=ip) + return templates.TemplateResponse( + "login.html", + {"request": request, "error": "Login fehlgeschlagen."}, + status_code=status.HTTP_401_UNAUTHORIZED, + ) + + response = RedirectResponse("/", status_code=303) + issue_session_cookie(response, username) + db.log_audit(username, "login_success", ip=ip) + return response + + +@app.post("/logout") +def logout(request: Request): + user = current_user(request) or "?" + response = RedirectResponse("/login", status_code=303) + clear_session_cookie(response) + db.log_audit(user, "logout", ip=client_ip(request)) + return response + + +# ----------------------------------------------------------------------------- +# Authenticated dashboard +# ----------------------------------------------------------------------------- + +@app.get("/", response_class=HTMLResponse) +def dashboard(request: Request): + user = current_user(request) + if not user: + return RedirectResponse("/login", status_code=303) + + allowed, reason = apply_allowed() + ctx = { + "request": request, + "user": user, + "csrf": issue_csrf_token(user), + "mode": db.get_state("mode") or "manual", + "kill_switch": db.get_state("kill_switch") == "1", + "circuit_open": db.get_state("apply_circuit_open") == "1", + "apply_failures": int(db.get_state("apply_recent_failures") or "0"), + "last_alert_heartbeat": db.get_state("last_alert_heartbeat") or "", + "apply_reachable": apply_client.health(), + "apply_allowed": allowed, + "apply_block_reason": reason, + "flats": db.recent_flats(50), + "applications": db.recent_applications(20), + "audit": db.recent_audit(15), + } + return templates.TemplateResponse("dashboard.html", ctx) + + +@app.get("/partials/dashboard", response_class=HTMLResponse) +def dashboard_partial(request: Request, user: str = Depends(require_user)): + """HTMX partial refresh — avoids leaking data to unauthenticated clients.""" + allowed, reason = apply_allowed() + ctx = { + "request": request, + "user": user, + "csrf": issue_csrf_token(user), + "mode": db.get_state("mode") or "manual", + "kill_switch": db.get_state("kill_switch") == "1", + "circuit_open": db.get_state("apply_circuit_open") == "1", + "apply_failures": int(db.get_state("apply_recent_failures") or "0"), + "last_alert_heartbeat": db.get_state("last_alert_heartbeat") or "", + "apply_reachable": apply_client.health(), + "apply_allowed": allowed, + "apply_block_reason": reason, + "flats": db.recent_flats(50), + "applications": db.recent_applications(20), + "audit": db.recent_audit(15), + } + return templates.TemplateResponse("_dashboard_body.html", ctx) + + +# ----------------------------------------------------------------------------- +# State-changing actions (require auth + CSRF) +# ----------------------------------------------------------------------------- + +@app.post("/actions/mode") +async def action_mode( + request: Request, + mode: str = Form(...), + csrf: str = Form(...), + user: str = Depends(require_user), +): + require_csrf(request, csrf) + if mode not in ("manual", "auto"): + raise HTTPException(400, "invalid mode") + db.set_state("mode", mode) + db.log_audit(user, "set_mode", mode, client_ip(request)) + return RedirectResponse("/", status_code=303) + + +@app.post("/actions/kill-switch") +async def action_kill_switch( + request: Request, + value: str = Form(...), + csrf: str = Form(...), + user: str = Depends(require_user), +): + require_csrf(request, csrf) + new = "1" if value == "on" else "0" + db.set_state("kill_switch", new) + db.log_audit(user, "set_kill_switch", new, client_ip(request)) + return RedirectResponse("/", status_code=303) + + +@app.post("/actions/reset-circuit") +async def action_reset_circuit( + request: Request, + csrf: str = Form(...), + user: str = Depends(require_user), +): + require_csrf(request, csrf) + db.set_state("apply_circuit_open", "0") + db.set_state("apply_recent_failures", "0") + db.log_audit(user, "reset_circuit", "", client_ip(request)) + return RedirectResponse("/", status_code=303) + + +@app.post("/actions/apply") +async def action_apply( + request: Request, + flat_id: str = Form(...), + csrf: str = Form(...), + user: str = Depends(require_user), +): + require_csrf(request, csrf) + allowed, reason = apply_allowed() + if not allowed: + raise HTTPException(409, f"apply disabled: {reason}") + + flat = db.get_flat(flat_id) + if not flat: + raise HTTPException(404, "flat not found") + + db.log_audit(user, "trigger_apply", f"flat_id={flat_id}", client_ip(request)) + # Run apply in background so the UI returns fast + asyncio.create_task(asyncio.to_thread(run_apply, flat_id, flat["link"], "user")) + return RedirectResponse("/", status_code=303) + + +# ----------------------------------------------------------------------------- +# Internal endpoints (called by alert/apply services) +# ----------------------------------------------------------------------------- + +@app.post("/internal/flats") +async def internal_submit_flat( + payload: dict, + _guard: None = Depends(require_internal), +): + if not payload.get("id") or not payload.get("link"): + raise HTTPException(400, "id and link required") + matched = matches_criteria(payload) + is_new = db.upsert_flat(payload, matched) + if not is_new: + return {"status": "duplicate"} + + if matched: + db.log_audit("alert", "flat_matched", f"id={payload['id']} rent={payload.get('total_rent')}") + if db.get_state("mode") == "auto": + allowed, reason = apply_allowed() + if allowed: + db.log_audit("system", "auto_apply", f"flat_id={payload['id']}") + asyncio.create_task(asyncio.to_thread(run_apply, str(payload["id"]), payload["link"], "auto")) + else: + db.log_audit("system", "auto_apply_blocked", reason) + return {"status": "ok", "matched": matched} + + +@app.post("/internal/heartbeat") +async def internal_heartbeat( + payload: dict, + _guard: None = Depends(require_internal), +): + service = payload.get("service", "unknown") + db.set_state(f"last_{service}_heartbeat", db.now_iso()) + return {"status": "ok"} diff --git a/web/apply_client.py b/web/apply_client.py new file mode 100644 index 0000000..4a4ea9a --- /dev/null +++ b/web/apply_client.py @@ -0,0 +1,34 @@ +import logging +import requests + +from settings import APPLY_URL, APPLY_TIMEOUT, INTERNAL_API_KEY + +logger = logging.getLogger("web") + + +class ApplyClient: + def __init__(self): + self.base = APPLY_URL.rstrip("/") + self.timeout = APPLY_TIMEOUT + self.headers = {"X-Internal-Api-Key": INTERNAL_API_KEY} + + def health(self) -> bool: + try: + r = requests.get(f"{self.base}/health", timeout=5) + return r.ok + except requests.RequestException: + return False + + def apply(self, url: str) -> dict: + try: + r = requests.post( + f"{self.base}/apply", + json={"url": url}, + headers=self.headers, + timeout=self.timeout, + ) + if r.status_code >= 400: + return {"success": False, "message": f"apply HTTP {r.status_code}: {r.text[:300]}"} + return r.json() + except requests.RequestException as e: + return {"success": False, "message": f"apply unreachable: {e}"} diff --git a/web/auth.py b/web/auth.py new file mode 100644 index 0000000..c93aca1 --- /dev/null +++ b/web/auth.py @@ -0,0 +1,134 @@ +import hmac +import secrets +import threading +import time +from typing import Optional + +from argon2 import PasswordHasher +from argon2.exceptions import VerifyMismatchError, InvalidHash +from fastapi import HTTPException, Request, Response, status +from itsdangerous import BadSignature, SignatureExpired, URLSafeTimedSerializer + +from settings import ( + AUTH_PASSWORD_HASH, + AUTH_USERNAME, + COOKIE_SECURE, + LOGIN_RATE_LIMIT, + LOGIN_RATE_WINDOW_SECONDS, + SESSION_COOKIE_NAME, + SESSION_MAX_AGE_SECONDS, + SESSION_SECRET, +) + +_hasher = PasswordHasher() +_serializer = URLSafeTimedSerializer(SESSION_SECRET, salt="session") +_csrf_serializer = URLSafeTimedSerializer(SESSION_SECRET, salt="csrf") + + +# ---------- Password & session ---------- + +def verify_password(username: str, password: str) -> bool: + if not hmac.compare_digest(username or "", AUTH_USERNAME): + # run hasher anyway to keep timing similar (and not leak whether user exists) + try: + _hasher.verify(AUTH_PASSWORD_HASH, password) + except Exception: + pass + return False + try: + _hasher.verify(AUTH_PASSWORD_HASH, password) + return True + except (VerifyMismatchError, InvalidHash): + return False + + +def issue_session_cookie(response: Response, username: str) -> None: + token = _serializer.dumps({"u": username, "iat": int(time.time())}) + response.set_cookie( + key=SESSION_COOKIE_NAME, + value=token, + max_age=SESSION_MAX_AGE_SECONDS, + httponly=True, + secure=COOKIE_SECURE, + samesite="strict", + path="/", + ) + + +def clear_session_cookie(response: Response) -> None: + response.delete_cookie( + SESSION_COOKIE_NAME, + path="/", + secure=COOKIE_SECURE, + httponly=True, + samesite="strict", + ) + + +def current_user(request: Request) -> Optional[str]: + token = request.cookies.get(SESSION_COOKIE_NAME) + if not token: + return None + try: + data = _serializer.loads(token, max_age=SESSION_MAX_AGE_SECONDS) + except (BadSignature, SignatureExpired): + return None + return data.get("u") + + +def require_user(request: Request) -> str: + user = current_user(request) + if not user: + raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="login required") + return user + + +# ---------- CSRF (synchronizer token bound to session) ---------- + +def issue_csrf_token(username: str) -> str: + return _csrf_serializer.dumps({"u": username}) + + +def verify_csrf(request: Request, submitted: str) -> bool: + user = current_user(request) + if not user or not submitted: + return False + try: + data = _csrf_serializer.loads(submitted, max_age=SESSION_MAX_AGE_SECONDS) + except (BadSignature, SignatureExpired): + return False + return hmac.compare_digest(str(data.get("u", "")), user) + + +def require_csrf(request: Request, token: str) -> None: + if not verify_csrf(request, token): + raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="bad csrf") + + +# ---------- Login rate limiting (in-memory, per IP) ---------- + +_rate_lock = threading.Lock() +_rate_log: dict[str, list[float]] = {} + + +def rate_limit_login(ip: str) -> bool: + """Returns True if the request is allowed.""" + now = time.time() + cutoff = now - LOGIN_RATE_WINDOW_SECONDS + with _rate_lock: + attempts = [t for t in _rate_log.get(ip, []) if t > cutoff] + if len(attempts) >= LOGIN_RATE_LIMIT: + _rate_log[ip] = attempts + return False + attempts.append(now) + _rate_log[ip] = attempts + # opportunistic cleanup + if len(_rate_log) > 1024: + for k in list(_rate_log.keys()): + if not _rate_log[k] or _rate_log[k][-1] < cutoff: + _rate_log.pop(k, None) + return True + + +def constant_time_compare(a: str, b: str) -> bool: + return hmac.compare_digest(a or "", b or "") diff --git a/web/db.py b/web/db.py new file mode 100644 index 0000000..ee22142 --- /dev/null +++ b/web/db.py @@ -0,0 +1,219 @@ +import json +import sqlite3 +import threading +from datetime import datetime, timezone +from typing import Any, Iterable + +from settings import DB_PATH + +_lock = threading.Lock() + + +def _connect() -> sqlite3.Connection: + conn = sqlite3.connect(DB_PATH, isolation_level=None, check_same_thread=False) + conn.row_factory = sqlite3.Row + conn.execute("PRAGMA journal_mode=WAL") + conn.execute("PRAGMA foreign_keys=ON") + return conn + + +_conn: sqlite3.Connection = _connect() + + +SCHEMA = """ +CREATE TABLE IF NOT EXISTS flats ( + id TEXT PRIMARY KEY, + link TEXT NOT NULL, + address TEXT, + rooms REAL, + size REAL, + total_rent REAL, + sqm_price REAL, + year_built TEXT, + wbs TEXT, + connectivity_morning_time REAL, + connectivity_night_time REAL, + address_link_gmaps TEXT, + payload_json TEXT NOT NULL, + matched_criteria INTEGER NOT NULL DEFAULT 0, + discovered_at TEXT NOT NULL +); + +CREATE INDEX IF NOT EXISTS idx_flats_discovered ON flats(discovered_at DESC); +CREATE INDEX IF NOT EXISTS idx_flats_matched ON flats(matched_criteria); + +CREATE TABLE IF NOT EXISTS applications ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + flat_id TEXT NOT NULL, + url TEXT NOT NULL, + triggered_by TEXT NOT NULL, -- 'user' | 'auto' + started_at TEXT NOT NULL, + finished_at TEXT, + success INTEGER, + message TEXT, + FOREIGN KEY (flat_id) REFERENCES flats(id) +); + +CREATE INDEX IF NOT EXISTS idx_applications_flat ON applications(flat_id); +CREATE INDEX IF NOT EXISTS idx_applications_started ON applications(started_at DESC); + +CREATE TABLE IF NOT EXISTS state ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS audit_log ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + timestamp TEXT NOT NULL, + actor TEXT NOT NULL, + action TEXT NOT NULL, + details TEXT, + ip TEXT +); + +CREATE INDEX IF NOT EXISTS idx_audit_timestamp ON audit_log(timestamp DESC); +""" + +DEFAULTS = { + "mode": "manual", # 'manual' | 'auto' + "kill_switch": "0", # '1' = apply disabled + "apply_circuit_open": "0", # '1' = opened by circuit breaker + "apply_recent_failures": "0", + "last_alert_heartbeat": "", + "last_apply_heartbeat": "", +} + + +def init_db() -> None: + with _lock: + _conn.executescript(SCHEMA) + for k, v in DEFAULTS.items(): + _conn.execute("INSERT OR IGNORE INTO state(key, value) VALUES (?, ?)", (k, v)) + + +def now_iso() -> str: + return datetime.now(timezone.utc).isoformat(timespec="seconds") + + +def get_state(key: str) -> str | None: + row = _conn.execute("SELECT value FROM state WHERE key = ?", (key,)).fetchone() + return row["value"] if row else None + + +def set_state(key: str, value: str) -> None: + with _lock: + _conn.execute( + "INSERT INTO state(key, value) VALUES (?, ?) " + "ON CONFLICT(key) DO UPDATE SET value = excluded.value", + (key, value), + ) + + +def upsert_flat(payload: dict, matched: bool) -> bool: + """Returns True if this flat is new.""" + flat_id = str(payload["id"]) + conn_info = payload.get("connectivity") or {} + with _lock: + existing = _conn.execute("SELECT id FROM flats WHERE id = ?", (flat_id,)).fetchone() + if existing: + return False + _conn.execute( + """ + INSERT INTO flats( + id, link, address, rooms, size, total_rent, sqm_price, year_built, wbs, + connectivity_morning_time, connectivity_night_time, address_link_gmaps, + payload_json, matched_criteria, discovered_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + flat_id, + payload.get("link", ""), + payload.get("address", ""), + payload.get("rooms"), + payload.get("size"), + payload.get("total_rent"), + payload.get("sqm_price"), + str(payload.get("year_built", "")), + str(payload.get("wbs", "")), + conn_info.get("morning_time"), + conn_info.get("night_time"), + payload.get("address_link_gmaps"), + json.dumps(payload, default=str), + 1 if matched else 0, + now_iso(), + ), + ) + return True + + +def recent_flats(limit: int = 50) -> list[sqlite3.Row]: + return list( + _conn.execute( + """ + SELECT f.*, + (SELECT success FROM applications a WHERE a.flat_id = f.id + ORDER BY a.started_at DESC LIMIT 1) AS last_application_success, + (SELECT message FROM applications a WHERE a.flat_id = f.id + ORDER BY a.started_at DESC LIMIT 1) AS last_application_message, + (SELECT started_at FROM applications a WHERE a.flat_id = f.id + ORDER BY a.started_at DESC LIMIT 1) AS last_application_at + FROM flats f + ORDER BY f.discovered_at DESC + LIMIT ? + """, + (limit,), + ).fetchall() + ) + + +def get_flat(flat_id: str) -> sqlite3.Row | None: + return _conn.execute("SELECT * FROM flats WHERE id = ?", (flat_id,)).fetchone() + + +def start_application(flat_id: str, url: str, triggered_by: str) -> int: + with _lock: + cur = _conn.execute( + "INSERT INTO applications(flat_id, url, triggered_by, started_at) VALUES (?, ?, ?, ?)", + (flat_id, url, triggered_by, now_iso()), + ) + return cur.lastrowid + + +def finish_application(app_id: int, success: bool, message: str) -> None: + with _lock: + _conn.execute( + "UPDATE applications SET finished_at = ?, success = ?, message = ? WHERE id = ?", + (now_iso(), 1 if success else 0, message, app_id), + ) + + +def recent_applications(limit: int = 20) -> list[sqlite3.Row]: + return list( + _conn.execute( + """ + SELECT a.*, f.address, f.link + FROM applications a + JOIN flats f ON f.id = a.flat_id + ORDER BY a.started_at DESC + LIMIT ? + """, + (limit,), + ).fetchall() + ) + + +def log_audit(actor: str, action: str, details: str = "", ip: str = "") -> None: + with _lock: + _conn.execute( + "INSERT INTO audit_log(timestamp, actor, action, details, ip) VALUES (?, ?, ?, ?, ?)", + (now_iso(), actor, action, details, ip), + ) + + +def recent_audit(limit: int = 30) -> list[sqlite3.Row]: + return list( + _conn.execute( + "SELECT * FROM audit_log ORDER BY timestamp DESC LIMIT ?", + (limit,), + ).fetchall() + ) diff --git a/web/requirements.txt b/web/requirements.txt new file mode 100644 index 0000000..56293ef --- /dev/null +++ b/web/requirements.txt @@ -0,0 +1,8 @@ +fastapi==0.115.5 +uvicorn[standard]==0.32.1 +jinja2==3.1.4 +argon2-cffi==23.1.0 +itsdangerous==2.2.0 +python-multipart==0.0.17 +python-dotenv==1.0.1 +requests==2.32.5 diff --git a/web/settings.py b/web/settings.py new file mode 100644 index 0000000..2f726b1 --- /dev/null +++ b/web/settings.py @@ -0,0 +1,54 @@ +import secrets +import sys +from os import getenv +from pathlib import Path + +from dotenv import load_dotenv + +load_dotenv() + + +def _required(key: str) -> str: + val = getenv(key) + if not val: + print(f"missing required env var: {key}", file=sys.stderr) + sys.exit(1) + return val + + +# --- Auth --- +AUTH_USERNAME: str = _required("AUTH_USERNAME") +# argon2 hash of the password. Generate via: +# python -c "from argon2 import PasswordHasher; print(PasswordHasher().hash(''))" +AUTH_PASSWORD_HASH: str = _required("AUTH_PASSWORD_HASH") + +# Signs session cookies. If missing -> ephemeral random secret (invalidates sessions on restart). +SESSION_SECRET: str = getenv("SESSION_SECRET") or secrets.token_urlsafe(48) +SESSION_COOKIE_NAME: str = "lazyflat_session" +SESSION_MAX_AGE_SECONDS: int = int(getenv("SESSION_MAX_AGE_SECONDS", str(60 * 60 * 24 * 7))) + +# When behind an HTTPS proxy (Coolify/Traefik) this MUST be true so cookies are Secure. +COOKIE_SECURE: bool = getenv("COOKIE_SECURE", "true").lower() in ("true", "1", "yes", "on") + +# --- Internal service auth --- +INTERNAL_API_KEY: str = _required("INTERNAL_API_KEY") + +# --- Apply service --- +APPLY_URL: str = getenv("APPLY_URL", "http://apply:8000") +APPLY_TIMEOUT: int = int(getenv("APPLY_TIMEOUT", "600")) +# Circuit breaker: disable auto-apply after N consecutive apply failures. +APPLY_FAILURE_THRESHOLD: int = int(getenv("APPLY_FAILURE_THRESHOLD", "3")) + +# --- Storage --- +DATA_DIR: Path = Path(getenv("DATA_DIR", "/data")) +DATA_DIR.mkdir(parents=True, exist_ok=True) +DB_PATH: Path = DATA_DIR / "lazyflat.sqlite" + +# --- Rate limiting --- +LOGIN_RATE_LIMIT: int = int(getenv("LOGIN_RATE_LIMIT", "5")) +LOGIN_RATE_WINDOW_SECONDS: int = int(getenv("LOGIN_RATE_WINDOW_SECONDS", "900")) + +# --- Filter criteria (mirrored from original flat-alert) --- +FILTER_ROOMS: list[float] = [float(r) for r in getenv("FILTER_ROOMS", "2.0,2.5").split(",") if r.strip()] +FILTER_MAX_RENT: float = float(getenv("FILTER_MAX_RENT", "1500")) +FILTER_MAX_MORNING_COMMUTE: float = float(getenv("FILTER_MAX_MORNING_COMMUTE", "50")) diff --git a/web/static/.gitkeep b/web/static/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/web/templates/_dashboard_body.html b/web/templates/_dashboard_body.html new file mode 100644 index 0000000..04f9251 --- /dev/null +++ b/web/templates/_dashboard_body.html @@ -0,0 +1,176 @@ +
+
+
alert
+
+ {% if last_alert_heartbeat %} + live + {% else %} + kein Heartbeat + {% endif %} +
+
letzter Heartbeat: {{ last_alert_heartbeat or "—" }}
+
+ +
+
apply
+
+ {% if apply_reachable %} + reachable + {% else %} + down + {% endif %} +
+
+ {% if circuit_open %} + circuit open + {% elif apply_failures > 0 %} + {{ apply_failures }} recent failure(s) + {% else %} + healthy + {% endif %} +
+
+ +
+
Modus
+
+ {% if mode == "auto" %} + full-auto + {% else %} + manuell + {% endif %} +
+
+ + + +
+
+ +
+
Kill‑Switch
+
+ {% if kill_switch %} + apply gestoppt + {% else %} + aktiv + {% endif %} +
+
+ + + +
+ {% if circuit_open %} +
+ + +
+ {% endif %} +
+
+ +{% if not apply_allowed %} +
+ apply blockiert + {{ apply_block_reason }} +
+{% endif %} + +
+
+

Wohnungen

+ {{ flats|length }} zuletzt gesehen +
+
+ {% for flat in flats %} +
+
+
+ + {{ flat.address or flat.link }} + + {% if flat.matched_criteria %} + match + {% else %} + info + {% endif %} + {% if flat.last_application_success == 1 %} + beworben + {% elif flat.last_application_success == 0 %} + apply fehlgeschlagen + {% endif %} +
+
+ {% if flat.rooms %}{{ "%.1f"|format(flat.rooms) }} Z{% endif %} + {% if flat.size %} · {{ "%.0f"|format(flat.size) }} m²{% endif %} + {% if flat.total_rent %} · {{ "%.0f"|format(flat.total_rent) }} €{% endif %} + {% if flat.sqm_price %} ({{ "%.2f"|format(flat.sqm_price) }} €/m²){% endif %} + {% if flat.connectivity_morning_time %} · {{ "%.0f"|format(flat.connectivity_morning_time) }} min morgens{% endif %} + · entdeckt {{ flat.discovered_at }} +
+ {% if flat.last_application_message %} +
↳ {{ flat.last_application_message }}
+ {% endif %} +
+
+ {% if apply_allowed and not flat.last_application_success %} +
+ + + +
+ {% endif %} +
+
+ {% else %} +
Noch keine Wohnungen gesehen.
+ {% endfor %} +
+
+ +
+
+

Letzte Bewerbungen

+
+ {% for a in applications %} +
+
+ {% if a.success == 1 %}ok + {% elif a.success == 0 %}fail + {% else %}läuft{% endif %} + {{ a.triggered_by }} + {{ a.started_at }} +
+
{{ a.address or a.url }}
+ {% if a.message %}
{{ a.message }}
{% endif %} +
+ {% else %} +
Keine Bewerbungen bisher.
+ {% endfor %} +
+
+ +
+

Audit-Log

+
+ {% for e in audit %} +
+ {{ e.timestamp }} + {{ e.actor }} + {{ e.action }} + {% if e.details %}— {{ e.details }}{% endif %} +
+ {% else %} +
leer
+ {% endfor %} +
+
+
diff --git a/web/templates/base.html b/web/templates/base.html new file mode 100644 index 0000000..f923bd3 --- /dev/null +++ b/web/templates/base.html @@ -0,0 +1,31 @@ + + + + + + + {% block title %}lazyflat{% endblock %} + + + + + + {% block body %}{% endblock %} + + diff --git a/web/templates/dashboard.html b/web/templates/dashboard.html new file mode 100644 index 0000000..2f406c8 --- /dev/null +++ b/web/templates/dashboard.html @@ -0,0 +1,25 @@ +{% extends "base.html" %} +{% block title %}lazyflat dashboard{% endblock %} +{% block body %} +
+
+
+
+

lazyflat

+
+
+ {{ user }} +
+ +
+
+
+
+ +
+
+ {% include "_dashboard_body.html" %} +
+
+{% endblock %} diff --git a/web/templates/login.html b/web/templates/login.html new file mode 100644 index 0000000..de74e42 --- /dev/null +++ b/web/templates/login.html @@ -0,0 +1,28 @@ +{% extends "base.html" %} +{% block title %}Login — lazyflat{% endblock %} +{% block body %} +
+
+

lazyflat

+

Anmeldung erforderlich

+ + {% if error %} +
{{ error }}
+ {% endif %} + +
+
+ + +
+
+ + +
+ +
+
+
+{% endblock %}