Files
mealie/mealie/services/backups/imports.py
Hayden b3573dc078 v0.2.0 (#143)
* Changed uvicorn port to 80

* Changed port in docker-compose to match dockerfile

* Readded environment variables in docker-compose

* production image rework

* Use opengraph metadata to make basic recipe cards when full recipe metadata is not available

* fixed instrucitons on parse

* add last_recipe

* automated testing

* roadmap update

* Sqlite (#75)

* file structure

* auto-test

* take 2

* refactor ap scheduler and startup process

* fixed scraper error

* database abstraction

* database abstraction

* port recipes over to new schema

* meal migration

* start settings migration

* finale mongo port

* backup improvements

* migration imports to new DB structure

* unused import cleanup

* docs strings

* settings and theme import logic

* cleanup

* fixed tinydb error

* requirements

* fuzzy search

* remove scratch file

* sqlalchemy models

* improved search ui

* recipe models almost done

* sql modal population

* del scratch

* rewrite database model mixins

* mostly grabage

* recipe updates

* working sqllite

* remove old files and reorganize

* final cleanup

Co-authored-by: Hayden <hay-kot@pm.me>

* Backup card (#78)

* backup / import dialog

* upgrade to new tag method

* New import card

* rename settings.py to app_config.py

* migrate to poetry for development

* fix failing test

Co-authored-by: Hayden <hay-kot@pm.me>

* added mkdocs to docker-compose

* Translations (#72)

* Translations + danish

* changed back proxy target to use ENV

* Resolved more merge conflicts

* Removed test in translation

* Documentation of translations

* Updated translations

* removed old packages

Co-authored-by: Hayden <64056131+hay-kot@users.noreply.github.com>

* fail to start bug fixes

* feature: prep/cook/total time slots (#80)

Co-authored-by: Hayden <hay-kot@pm.me>

* missing bind attributes

* Bug fixes (#81)

* fix: url remains after succesful import

* docs: changelog + update todos

* arm image

* arm compose

* compose updates

* update poetry

* arm support

Co-authored-by: Hayden <hay-kot@pm.me>

* dockerfile hotfix

* dockerfile hotfix

* Version Release Final Touches (#84)

* Remove slim

* bug: opacity issues

* bug: startup failure with no database

* ci/cd on dev branch

* formatting

* v0.1.0 documentation

Co-authored-by: Hayden <hay-kot@pm.me>

* db init hotfix

* bug: fix crash in mongo

* fix mongo bug

* fixed version notifier

* finale changelog

* Dropping Mongo From Dev Branch (#89)

* Fix link to Docker Hub

Found an extra s. DESTROYED it.

* initial pass

* second pass cleanup

* backup card framework

* backup card functionality

* translation

* upload button vile creation

* Release v0.1.0 Candidate (#85)

* Changed uvicorn port to 80

* Changed port in docker-compose to match dockerfile

* Readded environment variables in docker-compose

* production image rework

* Use opengraph metadata to make basic recipe cards when full recipe metadata is not available

* fixed instrucitons on parse

* add last_recipe

* automated testing

* roadmap update

* Sqlite (#75)

* file structure

* auto-test

* take 2

* refactor ap scheduler and startup process

* fixed scraper error

* database abstraction

* database abstraction

* port recipes over to new schema

* meal migration

* start settings migration

* finale mongo port

* backup improvements

* migration imports to new DB structure

* unused import cleanup

* docs strings

* settings and theme import logic

* cleanup

* fixed tinydb error

* requirements

* fuzzy search

* remove scratch file

* sqlalchemy models

* improved search ui

* recipe models almost done

* sql modal population

* del scratch

* rewrite database model mixins

* mostly grabage

* recipe updates

* working sqllite

* remove old files and reorganize

* final cleanup

Co-authored-by: Hayden <hay-kot@pm.me>

* Backup card (#78)

* backup / import dialog

* upgrade to new tag method

* New import card

* rename settings.py to app_config.py

* migrate to poetry for development

* fix failing test

Co-authored-by: Hayden <hay-kot@pm.me>

* added mkdocs to docker-compose

* Translations (#72)

* Translations + danish

* changed back proxy target to use ENV

* Resolved more merge conflicts

* Removed test in translation

* Documentation of translations

* Updated translations

* removed old packages

Co-authored-by: Hayden <64056131+hay-kot@users.noreply.github.com>

* fail to start bug fixes

* feature: prep/cook/total time slots (#80)

Co-authored-by: Hayden <hay-kot@pm.me>

* missing bind attributes

* Bug fixes (#81)

* fix: url remains after succesful import

* docs: changelog + update todos

* arm image

* arm compose

* compose updates

* update poetry

* arm support

Co-authored-by: Hayden <hay-kot@pm.me>

* dockerfile hotfix

* dockerfile hotfix

* Version Release Final Touches (#84)

* Remove slim

* bug: opacity issues

* bug: startup failure with no database

* ci/cd on dev branch

* formatting

* v0.1.0 documentation

Co-authored-by: Hayden <hay-kot@pm.me>

* db init hotfix

* bug: fix crash in mongo

* fix mongo bug

* fixed version notifier

* finale changelog

Co-authored-by: kentora <=>
Co-authored-by: Hayden <hay-kot@pm.me>
Co-authored-by: Richard Mitic <richard.h.mitic@gmail.com>
Co-authored-by: kentora <kentora@kentora.dk>

* build container

* webscraper hotfix

* dev bug: change data location to prevent reloads

* api docs

* api docs bug

* workflow update

Co-authored-by: David Young <davidy@funkypenguin.co.nz>
Co-authored-by: Hayden <hay-kot@pm.me>
Co-authored-by: Richard Mitic <richard.h.mitic@gmail.com>
Co-authored-by: kentora <kentora@kentora.dk>

* Add French Translation (#93)

* New tests (#94)

* dev-bug: fixed vscode freezes

* test: refactor database init to support tests

Co-authored-by: Hayden <hay-kot@pm.me>

* Mealplan CRUD Tests (#95)

* dev-bug: fixed vscode freezes

* test: refactor database init to support tests

* mealplan CRUD testing

Co-authored-by: Hayden <hay-kot@pm.me>

* Fix typos (#96)

* Settings, Themes and Migration Route Tests (#100)

* dev-bug: fixed vscode freezes

* test: refactor database init to support tests

* mealplan CRUD testing

* restructure test folder

* git attributes

* tests: migration, settings, theme routes testing

Co-authored-by: Hayden <hay-kot@pm.me>

* Refactor + New Docker File (#105)

* dev-bug: fixed vscode freezes

* test: refactor database init to support tests

* mealplan CRUD testing

* restructure test folder

* git attributes

* tests: migration, settings, theme routes testing

* docker-file shrink

* rebuild

* refactor: moving directories around

* adding funding

Co-authored-by: Hayden <hay-kot@pm.me>

* Meal planner improvements (#107)

* dev-bug: fixed vscode freezes

* test: refactor database init to support tests

* mealplan CRUD testing

* restructure test folder

* git attributes

* tests: migration, settings, theme routes testing

* docker-file shrink

* rebuild

* refactor: moving directories around

* adding funding

* mealplan redesign

Co-authored-by: Hayden <hay-kot@pm.me>

* Upload component (#108)

* unified upload button + download backups

* javascript toolings

* fix vuetur config

* fixed type check error

* refactor: clean up bag javascript

Co-authored-by: Hayden <hay-kot@pm.me>

* Upload component (#113)

* unified upload button + download backups

* javascript toolings

* fix vuetur config

* fixed type check error

* refactor: clean up bag javascript

* UI updates + name validation

* docs: changelog + sp

* fixed route links

* changelog

Co-authored-by: Hayden <hay-kot@pm.me>

* fixed menu links

* fixed poetry install on docker.dev build

* Migration redesign (#119)

* migration redesign init

* new color picker

* changelog

* added UI language selection

* fix layout issue on recipe editor

* remove git as dependency

* added UI editor for original URL

* CI/CD Tests

* test: fixed migration routes

Co-authored-by: Hayden <hay-kot@pm.me>

* Fix link to dev-notes.md (#110)

* translation: add swedish (#128)

* language: da is Danish

* translations: add swedish

* scraper: unescape html in instructions (#129)

Some urls erroneously deliver escaped html their instructions,
sometimes they are even escaped on multiple levels like here:

https://www.ica.se/recept/kladdig-kladdkaka-722982/

```
>>> normalize_instruction("S&amp;auml;tt ugnen p&amp;aring; 200&amp;deg;C.")
'Sätt ugnen på 200°C.'
```

* v0.2.0 Updates (#130)

* migration redesign init

* new color picker

* changelog

* added UI language selection

* fix layout issue on recipe editor

* remove git as dependency

* added UI editor for original URL

* CI/CD Tests

* test: fixed migration routes

* test todos

* bug/added docker volume

* chowdow test data

* partial image recipe image testing

* added card section card

* settings form

* homepage cetegory ui

* frontend category placeholder

* fixed broken scheduler

* remove old files

* removed temp test

Co-authored-by: Hayden <hay-kot@pm.me>

* Fix missing translations key (#133)

* translation: add simplified & traditional chinese

* Fix missing translations

* fix chinese translations

* v0.2.0 Release Candidate (#141)

* Fix link to Docker Hub

Found an extra s. DESTROYED it.

* Release v0.1.0 Candidate (#85)

* Changed uvicorn port to 80

* Changed port in docker-compose to match dockerfile

* Readded environment variables in docker-compose

* production image rework

* Use opengraph metadata to make basic recipe cards when full recipe metadata is not available

* fixed instrucitons on parse

* add last_recipe

* automated testing

* roadmap update

* Sqlite (#75)

* file structure

* auto-test

* take 2

* refactor ap scheduler and startup process

* fixed scraper error

* database abstraction

* database abstraction

* port recipes over to new schema

* meal migration

* start settings migration

* finale mongo port

* backup improvements

* migration imports to new DB structure

* unused import cleanup

* docs strings

* settings and theme import logic

* cleanup

* fixed tinydb error

* requirements

* fuzzy search

* remove scratch file

* sqlalchemy models

* improved search ui

* recipe models almost done

* sql modal population

* del scratch

* rewrite database model mixins

* mostly grabage

* recipe updates

* working sqllite

* remove old files and reorganize

* final cleanup

Co-authored-by: Hayden <hay-kot@pm.me>

* Backup card (#78)

* backup / import dialog

* upgrade to new tag method

* New import card

* rename settings.py to app_config.py

* migrate to poetry for development

* fix failing test

Co-authored-by: Hayden <hay-kot@pm.me>

* added mkdocs to docker-compose

* Translations (#72)

* Translations + danish

* changed back proxy target to use ENV

* Resolved more merge conflicts

* Removed test in translation

* Documentation of translations

* Updated translations

* removed old packages

Co-authored-by: Hayden <64056131+hay-kot@users.noreply.github.com>

* fail to start bug fixes

* feature: prep/cook/total time slots (#80)

Co-authored-by: Hayden <hay-kot@pm.me>

* missing bind attributes

* Bug fixes (#81)

* fix: url remains after succesful import

* docs: changelog + update todos

* arm image

* arm compose

* compose updates

* update poetry

* arm support

Co-authored-by: Hayden <hay-kot@pm.me>

* dockerfile hotfix

* dockerfile hotfix

* Version Release Final Touches (#84)

* Remove slim

* bug: opacity issues

* bug: startup failure with no database

* ci/cd on dev branch

* formatting

* v0.1.0 documentation

Co-authored-by: Hayden <hay-kot@pm.me>

* db init hotfix

* bug: fix crash in mongo

* fix mongo bug

* fixed version notifier

* finale changelog

Co-authored-by: kentora <=>
Co-authored-by: Hayden <hay-kot@pm.me>
Co-authored-by: Richard Mitic <richard.h.mitic@gmail.com>
Co-authored-by: kentora <kentora@kentora.dk>

* build container

* webscraper hotfix

* notes hot fix

* bug: mongo updates fail #99

* Fix error message (#101)

* gh funding

* Create Issue Templates (#125)

* Create bug_report.md

* Create config.yml

Included a link to feature requests.

* Update config.yml

Fixed link I had for testing to the actual link

* Update bug_report.md

fix capitalization

* Update .github/ISSUE_TEMPLATE/bug_report.md

Co-authored-by: Stephen Brown II <Stephen.Brown2@gmail.com>

Co-authored-by: Stephen Brown II <Stephen.Brown2@gmail.com>

* merge kentors changes

* refactor/recipe routers

* category/tag database relationship and endpoints

* frontend category management

* update branch todos

* bug/normalize recipe steps html

* remove console.log +  refactor categories

* fix categories database errors

* refactor/ router endpoint

* refactor/ remove old code

* drag and drop ingredients

* general cleanup

* route refactoring

* changelog

* api refactoring + random cleanup

* fixed backwards sort

* Update mkdocs.yml (#137)

Fix warning from Deploy Docs github action

* fixed navigate on enter in search

* refactor/create global css

* added category scroll

* cleanup todos

* debug routes

* docs/new gifs & general updates

* cleanup

* fix list test

Co-authored-by: David Young <davidy@funkypenguin.co.nz>
Co-authored-by: Hayden <hay-kot@pm.me>
Co-authored-by: Richard Mitic <richard.h.mitic@gmail.com>
Co-authored-by: kentora <kentora@kentora.dk>
Co-authored-by: Alexei Pesic <pesic.alexei@gmail.com>
Co-authored-by: Andrew <dpieski@gmail.com>
Co-authored-by: Stephen Brown II <Stephen.Brown2@gmail.com>

* fix build

* fix duplicate editor

* fixed docker mount problem

* python 3.9

* added tasks for non-docker development

* remove old scripts

* dev updates

* fixed no image upload option

* get version from backend

* final docs pass

* .gitignore

Co-authored-by: kentora <=>
Co-authored-by: Hayden <hay-kot@pm.me>
Co-authored-by: Richard Mitic <richard.h.mitic@gmail.com>
Co-authored-by: kentora <kentora@kentora.dk>
Co-authored-by: David Young <davidy@funkypenguin.co.nz>
Co-authored-by: Bastien <43323819+Batgame@users.noreply.github.com>
Co-authored-by: sephrat <34862846+sephrat@users.noreply.github.com>
Co-authored-by: Nick CJ <17556895+nickcj931@users.noreply.github.com>
Co-authored-by: dekvall <dkvldev@gmail.com>
Co-authored-by: wengtad <wengtad93@gmail.com>
Co-authored-by: Alexei Pesic <pesic.alexei@gmail.com>
Co-authored-by: Andrew <dpieski@gmail.com>
Co-authored-by: Stephen Brown II <Stephen.Brown2@gmail.com>
2021-02-08 09:47:40 -09:00

136 lines
4.7 KiB
Python

import json
import shutil
import zipfile
from pathlib import Path
from typing import List
from app_config import BACKUP_DIR, IMG_DIR, TEMP_DIR
from services.recipe_services import Recipe
from services.settings_services import SiteSettings, SiteTheme
from sqlalchemy.orm.session import Session
from utils.logger import logger
class ImportDatabase:
def __init__(
self,
session: Session,
zip_archive: str,
import_recipes: bool = True,
import_settings: bool = True,
import_themes: bool = True,
force_import: bool = False,
rebase: bool = False,
) -> None:
"""Import a database.zip file exported from mealie.
Args:
zip_archive (str): The filename contained in the backups directory
import_recipes (bool, optional): Import Recipes?. Defaults to True.
import_settings (bool, optional): Determines if settings are imported. Defaults to True.
import_themes (bool, optional): Determines if themes are imported. Defaults to True.
force_import (bool, optional): Force import will update all existing recipes. If False existing recipes are skipped. Defaults to False.
rebase (bool, optional): Rebase will first clear the database and then import Recipes. Defaults to False.
Raises:
Exception: If the zip file does not exists an exception raise.
"""
self.session = session
self.archive = BACKUP_DIR.joinpath(zip_archive)
self.imp_recipes = import_recipes
self.imp_settings = import_settings
self.imp_themes = import_themes
self.force_imports = force_import
self.force_rebase = rebase
if self.archive.is_file():
self.import_dir = TEMP_DIR.joinpath("active_import")
self.import_dir.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(self.archive, "r") as zip_ref:
zip_ref.extractall(self.import_dir)
pass
else:
raise Exception("Import file does not exist")
def run(self):
if self.imp_recipes:
report = self.import_recipes()
if self.imp_settings:
self.import_settings()
if self.imp_themes:
self.import_themes()
self.clean_up()
return report if report else None
def import_recipes(self):
recipe_dir: Path = self.import_dir.joinpath("recipes")
successful_imports = []
failed_imports = []
for recipe in recipe_dir.glob("*.json"):
with open(recipe, "r") as f:
recipe_dict = json.loads(f.read())
recipe_dict = ImportDatabase._recipe_migration(recipe_dict)
try:
recipe_obj = Recipe(**recipe_dict)
recipe_obj.save_to_db(self.session)
successful_imports.append(recipe.stem)
logger.info(f"Imported: {recipe.stem}")
except Exception as inst:
logger.error(inst)
logger.info(f"Failed Import: {recipe.stem}")
failed_imports.append(recipe.stem)
self._import_images(successful_imports)
return {"successful": successful_imports, "failed": failed_imports}
@staticmethod
def _recipe_migration(recipe_dict: dict) -> dict:
try:
del recipe_dict["_id"]
del recipe_dict["dateAdded"]
except:
logger.info("Detected new backup Schema, skipping migration...")
return recipe_dict
# Migration from list to Object Type Data
if type(recipe_dict["extras"]) == list:
recipe_dict["extras"] = {}
return recipe_dict
def _import_images(self, successful_imports: List[str]):
image_dir = self.import_dir.joinpath("images")
for image in image_dir.iterdir():
if image.stem in successful_imports:
shutil.copy(image, IMG_DIR)
def import_themes(self):
themes_file = self.import_dir.joinpath("themes", "themes.json")
with open(themes_file, "r") as f:
themes: list = json.loads(f.read())
for theme in themes:
new_theme = SiteTheme(**theme)
try:
new_theme.save_to_db(self.session)
except:
logger.info(f"Unable Import Theme {new_theme.name}")
def import_settings(self):
settings_file = self.import_dir.joinpath("settings", "settings.json")
with open(settings_file, "r") as f:
settings: dict = json.loads(f.read())
settings = SiteSettings(**settings)
settings.update(self.session)
def clean_up(self):
shutil.rmtree(TEMP_DIR)