mirror of
https://github.com/mealie-recipes/mealie.git
synced 2026-02-08 17:03:11 -05:00
v0.2.0 (#143)
* Changed uvicorn port to 80 * Changed port in docker-compose to match dockerfile * Readded environment variables in docker-compose * production image rework * Use opengraph metadata to make basic recipe cards when full recipe metadata is not available * fixed instrucitons on parse * add last_recipe * automated testing * roadmap update * Sqlite (#75) * file structure * auto-test * take 2 * refactor ap scheduler and startup process * fixed scraper error * database abstraction * database abstraction * port recipes over to new schema * meal migration * start settings migration * finale mongo port * backup improvements * migration imports to new DB structure * unused import cleanup * docs strings * settings and theme import logic * cleanup * fixed tinydb error * requirements * fuzzy search * remove scratch file * sqlalchemy models * improved search ui * recipe models almost done * sql modal population * del scratch * rewrite database model mixins * mostly grabage * recipe updates * working sqllite * remove old files and reorganize * final cleanup Co-authored-by: Hayden <hay-kot@pm.me> * Backup card (#78) * backup / import dialog * upgrade to new tag method * New import card * rename settings.py to app_config.py * migrate to poetry for development * fix failing test Co-authored-by: Hayden <hay-kot@pm.me> * added mkdocs to docker-compose * Translations (#72) * Translations + danish * changed back proxy target to use ENV * Resolved more merge conflicts * Removed test in translation * Documentation of translations * Updated translations * removed old packages Co-authored-by: Hayden <64056131+hay-kot@users.noreply.github.com> * fail to start bug fixes * feature: prep/cook/total time slots (#80) Co-authored-by: Hayden <hay-kot@pm.me> * missing bind attributes * Bug fixes (#81) * fix: url remains after succesful import * docs: changelog + update todos * arm image * arm compose * compose updates * update poetry * arm support Co-authored-by: Hayden <hay-kot@pm.me> * dockerfile hotfix * dockerfile hotfix * Version Release Final Touches (#84) * Remove slim * bug: opacity issues * bug: startup failure with no database * ci/cd on dev branch * formatting * v0.1.0 documentation Co-authored-by: Hayden <hay-kot@pm.me> * db init hotfix * bug: fix crash in mongo * fix mongo bug * fixed version notifier * finale changelog * Dropping Mongo From Dev Branch (#89) * Fix link to Docker Hub Found an extra s. DESTROYED it. * initial pass * second pass cleanup * backup card framework * backup card functionality * translation * upload button vile creation * Release v0.1.0 Candidate (#85) * Changed uvicorn port to 80 * Changed port in docker-compose to match dockerfile * Readded environment variables in docker-compose * production image rework * Use opengraph metadata to make basic recipe cards when full recipe metadata is not available * fixed instrucitons on parse * add last_recipe * automated testing * roadmap update * Sqlite (#75) * file structure * auto-test * take 2 * refactor ap scheduler and startup process * fixed scraper error * database abstraction * database abstraction * port recipes over to new schema * meal migration * start settings migration * finale mongo port * backup improvements * migration imports to new DB structure * unused import cleanup * docs strings * settings and theme import logic * cleanup * fixed tinydb error * requirements * fuzzy search * remove scratch file * sqlalchemy models * improved search ui * recipe models almost done * sql modal population * del scratch * rewrite database model mixins * mostly grabage * recipe updates * working sqllite * remove old files and reorganize * final cleanup Co-authored-by: Hayden <hay-kot@pm.me> * Backup card (#78) * backup / import dialog * upgrade to new tag method * New import card * rename settings.py to app_config.py * migrate to poetry for development * fix failing test Co-authored-by: Hayden <hay-kot@pm.me> * added mkdocs to docker-compose * Translations (#72) * Translations + danish * changed back proxy target to use ENV * Resolved more merge conflicts * Removed test in translation * Documentation of translations * Updated translations * removed old packages Co-authored-by: Hayden <64056131+hay-kot@users.noreply.github.com> * fail to start bug fixes * feature: prep/cook/total time slots (#80) Co-authored-by: Hayden <hay-kot@pm.me> * missing bind attributes * Bug fixes (#81) * fix: url remains after succesful import * docs: changelog + update todos * arm image * arm compose * compose updates * update poetry * arm support Co-authored-by: Hayden <hay-kot@pm.me> * dockerfile hotfix * dockerfile hotfix * Version Release Final Touches (#84) * Remove slim * bug: opacity issues * bug: startup failure with no database * ci/cd on dev branch * formatting * v0.1.0 documentation Co-authored-by: Hayden <hay-kot@pm.me> * db init hotfix * bug: fix crash in mongo * fix mongo bug * fixed version notifier * finale changelog Co-authored-by: kentora <=> Co-authored-by: Hayden <hay-kot@pm.me> Co-authored-by: Richard Mitic <richard.h.mitic@gmail.com> Co-authored-by: kentora <kentora@kentora.dk> * build container * webscraper hotfix * dev bug: change data location to prevent reloads * api docs * api docs bug * workflow update Co-authored-by: David Young <davidy@funkypenguin.co.nz> Co-authored-by: Hayden <hay-kot@pm.me> Co-authored-by: Richard Mitic <richard.h.mitic@gmail.com> Co-authored-by: kentora <kentora@kentora.dk> * Add French Translation (#93) * New tests (#94) * dev-bug: fixed vscode freezes * test: refactor database init to support tests Co-authored-by: Hayden <hay-kot@pm.me> * Mealplan CRUD Tests (#95) * dev-bug: fixed vscode freezes * test: refactor database init to support tests * mealplan CRUD testing Co-authored-by: Hayden <hay-kot@pm.me> * Fix typos (#96) * Settings, Themes and Migration Route Tests (#100) * dev-bug: fixed vscode freezes * test: refactor database init to support tests * mealplan CRUD testing * restructure test folder * git attributes * tests: migration, settings, theme routes testing Co-authored-by: Hayden <hay-kot@pm.me> * Refactor + New Docker File (#105) * dev-bug: fixed vscode freezes * test: refactor database init to support tests * mealplan CRUD testing * restructure test folder * git attributes * tests: migration, settings, theme routes testing * docker-file shrink * rebuild * refactor: moving directories around * adding funding Co-authored-by: Hayden <hay-kot@pm.me> * Meal planner improvements (#107) * dev-bug: fixed vscode freezes * test: refactor database init to support tests * mealplan CRUD testing * restructure test folder * git attributes * tests: migration, settings, theme routes testing * docker-file shrink * rebuild * refactor: moving directories around * adding funding * mealplan redesign Co-authored-by: Hayden <hay-kot@pm.me> * Upload component (#108) * unified upload button + download backups * javascript toolings * fix vuetur config * fixed type check error * refactor: clean up bag javascript Co-authored-by: Hayden <hay-kot@pm.me> * Upload component (#113) * unified upload button + download backups * javascript toolings * fix vuetur config * fixed type check error * refactor: clean up bag javascript * UI updates + name validation * docs: changelog + sp * fixed route links * changelog Co-authored-by: Hayden <hay-kot@pm.me> * fixed menu links * fixed poetry install on docker.dev build * Migration redesign (#119) * migration redesign init * new color picker * changelog * added UI language selection * fix layout issue on recipe editor * remove git as dependency * added UI editor for original URL * CI/CD Tests * test: fixed migration routes Co-authored-by: Hayden <hay-kot@pm.me> * Fix link to dev-notes.md (#110) * translation: add swedish (#128) * language: da is Danish * translations: add swedish * scraper: unescape html in instructions (#129) Some urls erroneously deliver escaped html their instructions, sometimes they are even escaped on multiple levels like here: https://www.ica.se/recept/kladdig-kladdkaka-722982/ ``` >>> normalize_instruction("S&auml;tt ugnen p&aring; 200&deg;C.") 'Sätt ugnen på 200°C.' ``` * v0.2.0 Updates (#130) * migration redesign init * new color picker * changelog * added UI language selection * fix layout issue on recipe editor * remove git as dependency * added UI editor for original URL * CI/CD Tests * test: fixed migration routes * test todos * bug/added docker volume * chowdow test data * partial image recipe image testing * added card section card * settings form * homepage cetegory ui * frontend category placeholder * fixed broken scheduler * remove old files * removed temp test Co-authored-by: Hayden <hay-kot@pm.me> * Fix missing translations key (#133) * translation: add simplified & traditional chinese * Fix missing translations * fix chinese translations * v0.2.0 Release Candidate (#141) * Fix link to Docker Hub Found an extra s. DESTROYED it. * Release v0.1.0 Candidate (#85) * Changed uvicorn port to 80 * Changed port in docker-compose to match dockerfile * Readded environment variables in docker-compose * production image rework * Use opengraph metadata to make basic recipe cards when full recipe metadata is not available * fixed instrucitons on parse * add last_recipe * automated testing * roadmap update * Sqlite (#75) * file structure * auto-test * take 2 * refactor ap scheduler and startup process * fixed scraper error * database abstraction * database abstraction * port recipes over to new schema * meal migration * start settings migration * finale mongo port * backup improvements * migration imports to new DB structure * unused import cleanup * docs strings * settings and theme import logic * cleanup * fixed tinydb error * requirements * fuzzy search * remove scratch file * sqlalchemy models * improved search ui * recipe models almost done * sql modal population * del scratch * rewrite database model mixins * mostly grabage * recipe updates * working sqllite * remove old files and reorganize * final cleanup Co-authored-by: Hayden <hay-kot@pm.me> * Backup card (#78) * backup / import dialog * upgrade to new tag method * New import card * rename settings.py to app_config.py * migrate to poetry for development * fix failing test Co-authored-by: Hayden <hay-kot@pm.me> * added mkdocs to docker-compose * Translations (#72) * Translations + danish * changed back proxy target to use ENV * Resolved more merge conflicts * Removed test in translation * Documentation of translations * Updated translations * removed old packages Co-authored-by: Hayden <64056131+hay-kot@users.noreply.github.com> * fail to start bug fixes * feature: prep/cook/total time slots (#80) Co-authored-by: Hayden <hay-kot@pm.me> * missing bind attributes * Bug fixes (#81) * fix: url remains after succesful import * docs: changelog + update todos * arm image * arm compose * compose updates * update poetry * arm support Co-authored-by: Hayden <hay-kot@pm.me> * dockerfile hotfix * dockerfile hotfix * Version Release Final Touches (#84) * Remove slim * bug: opacity issues * bug: startup failure with no database * ci/cd on dev branch * formatting * v0.1.0 documentation Co-authored-by: Hayden <hay-kot@pm.me> * db init hotfix * bug: fix crash in mongo * fix mongo bug * fixed version notifier * finale changelog Co-authored-by: kentora <=> Co-authored-by: Hayden <hay-kot@pm.me> Co-authored-by: Richard Mitic <richard.h.mitic@gmail.com> Co-authored-by: kentora <kentora@kentora.dk> * build container * webscraper hotfix * notes hot fix * bug: mongo updates fail #99 * Fix error message (#101) * gh funding * Create Issue Templates (#125) * Create bug_report.md * Create config.yml Included a link to feature requests. * Update config.yml Fixed link I had for testing to the actual link * Update bug_report.md fix capitalization * Update .github/ISSUE_TEMPLATE/bug_report.md Co-authored-by: Stephen Brown II <Stephen.Brown2@gmail.com> Co-authored-by: Stephen Brown II <Stephen.Brown2@gmail.com> * merge kentors changes * refactor/recipe routers * category/tag database relationship and endpoints * frontend category management * update branch todos * bug/normalize recipe steps html * remove console.log + refactor categories * fix categories database errors * refactor/ router endpoint * refactor/ remove old code * drag and drop ingredients * general cleanup * route refactoring * changelog * api refactoring + random cleanup * fixed backwards sort * Update mkdocs.yml (#137) Fix warning from Deploy Docs github action * fixed navigate on enter in search * refactor/create global css * added category scroll * cleanup todos * debug routes * docs/new gifs & general updates * cleanup * fix list test Co-authored-by: David Young <davidy@funkypenguin.co.nz> Co-authored-by: Hayden <hay-kot@pm.me> Co-authored-by: Richard Mitic <richard.h.mitic@gmail.com> Co-authored-by: kentora <kentora@kentora.dk> Co-authored-by: Alexei Pesic <pesic.alexei@gmail.com> Co-authored-by: Andrew <dpieski@gmail.com> Co-authored-by: Stephen Brown II <Stephen.Brown2@gmail.com> * fix build * fix duplicate editor * fixed docker mount problem * python 3.9 * added tasks for non-docker development * remove old scripts * dev updates * fixed no image upload option * get version from backend * final docs pass * .gitignore Co-authored-by: kentora <=> Co-authored-by: Hayden <hay-kot@pm.me> Co-authored-by: Richard Mitic <richard.h.mitic@gmail.com> Co-authored-by: kentora <kentora@kentora.dk> Co-authored-by: David Young <davidy@funkypenguin.co.nz> Co-authored-by: Bastien <43323819+Batgame@users.noreply.github.com> Co-authored-by: sephrat <34862846+sephrat@users.noreply.github.com> Co-authored-by: Nick CJ <17556895+nickcj931@users.noreply.github.com> Co-authored-by: dekvall <dkvldev@gmail.com> Co-authored-by: wengtad <wengtad93@gmail.com> Co-authored-by: Alexei Pesic <pesic.alexei@gmail.com> Co-authored-by: Andrew <dpieski@gmail.com> Co-authored-by: Stephen Brown II <Stephen.Brown2@gmail.com>
This commit is contained in:
Hayden
GitHub
@@ -3,16 +3,17 @@ import shutil
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from app_config import BACKUP_DIR, IMG_DIR, TEMP_DIR, TEMPLATE_DIR
|
||||
from db.db_setup import create_session
|
||||
from jinja2 import Template
|
||||
from services.meal_services import MealPlan
|
||||
from services.recipe_services import Recipe
|
||||
from services.settings_services import SiteSettings, SiteTheme
|
||||
from app_config import BACKUP_DIR, IMG_DIR, TEMP_DIR, TEMPLATE_DIR
|
||||
from utils.logger import logger
|
||||
|
||||
|
||||
class ExportDatabase:
|
||||
def __init__(self, tag=None, templates=None) -> None:
|
||||
def __init__(self, session, tag=None, templates=None) -> None:
|
||||
"""Export a Mealie database. Export interacts directly with class objects and can be used
|
||||
with any supported backend database platform. By default tags are timestands, and no Jinja2 templates are rendered
|
||||
|
||||
@@ -26,6 +27,7 @@ class ExportDatabase:
|
||||
else:
|
||||
export_tag = datetime.now().strftime("%Y-%b-%d")
|
||||
|
||||
self.session = session
|
||||
self.main_dir = TEMP_DIR.joinpath(export_tag)
|
||||
self.img_dir = self.main_dir.joinpath("images")
|
||||
self.recipe_dir = self.main_dir.joinpath("recipes")
|
||||
@@ -54,7 +56,7 @@ class ExportDatabase:
|
||||
dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def export_recipes(self):
|
||||
all_recipes = Recipe.get_all()
|
||||
all_recipes = Recipe.get_all(self.session)
|
||||
|
||||
for recipe in all_recipes:
|
||||
logger.info(f"Backing Up Recipes: {recipe}")
|
||||
@@ -86,12 +88,12 @@ class ExportDatabase:
|
||||
shutil.copy(file, self.img_dir.joinpath(file.name))
|
||||
|
||||
def export_settings(self):
|
||||
all_settings = SiteSettings.get_site_settings()
|
||||
all_settings = SiteSettings.get_site_settings(self.session)
|
||||
out_file = self.settings_dir.joinpath("settings.json")
|
||||
ExportDatabase._write_json_file(all_settings.dict(), out_file)
|
||||
|
||||
def export_themes(self):
|
||||
all_themes = SiteTheme.get_all()
|
||||
all_themes = SiteTheme.get_all(self.session)
|
||||
if all_themes:
|
||||
all_themes = [x.dict() for x in all_themes]
|
||||
out_file = self.themes_dir.joinpath("themes.json")
|
||||
@@ -100,7 +102,7 @@ class ExportDatabase:
|
||||
def export_meals(
|
||||
self,
|
||||
): #! Problem Parseing Datetime Objects... May come back to this
|
||||
meal_plans = MealPlan.get_all()
|
||||
meal_plans = MealPlan.get_all(self.session)
|
||||
if meal_plans:
|
||||
meal_plans = [x.dict() for x in meal_plans]
|
||||
|
||||
@@ -123,15 +125,27 @@ class ExportDatabase:
|
||||
return str(zip_path.absolute()) + ".zip"
|
||||
|
||||
|
||||
def backup_all(tag=None, templates=None):
|
||||
db_export = ExportDatabase(tag=tag, templates=templates)
|
||||
def backup_all(
|
||||
session,
|
||||
tag=None,
|
||||
templates=None,
|
||||
export_recipes=True,
|
||||
export_settings=True,
|
||||
export_themes=True,
|
||||
):
|
||||
db_export = ExportDatabase(session=session, tag=tag, templates=templates)
|
||||
|
||||
if export_recipes:
|
||||
db_export.export_recipes()
|
||||
db_export.export_images()
|
||||
|
||||
if export_settings:
|
||||
db_export.export_settings()
|
||||
|
||||
if export_themes:
|
||||
db_export.export_themes()
|
||||
# db_export.export_meals()
|
||||
|
||||
db_export.export_recipes()
|
||||
db_export.export_images()
|
||||
db_export.export_settings()
|
||||
db_export.export_themes()
|
||||
db_export.export_meals()
|
||||
#
|
||||
return db_export.finish_export()
|
||||
|
||||
|
||||
@@ -143,5 +157,6 @@ def auto_backup_job():
|
||||
for template in TEMPLATE_DIR.iterdir():
|
||||
templates.append(template)
|
||||
|
||||
backup_all(tag="Auto", templates=templates)
|
||||
session = create_session()
|
||||
backup_all(session=session, tag="Auto", templates=templates)
|
||||
logger.info("Auto Backup Called")
|
||||
|
||||
@@ -7,12 +7,14 @@ from typing import List
|
||||
from app_config import BACKUP_DIR, IMG_DIR, TEMP_DIR
|
||||
from services.recipe_services import Recipe
|
||||
from services.settings_services import SiteSettings, SiteTheme
|
||||
from sqlalchemy.orm.session import Session
|
||||
from utils.logger import logger
|
||||
|
||||
|
||||
class ImportDatabase:
|
||||
def __init__(
|
||||
self,
|
||||
session: Session,
|
||||
zip_archive: str,
|
||||
import_recipes: bool = True,
|
||||
import_settings: bool = True,
|
||||
@@ -33,7 +35,7 @@ class ImportDatabase:
|
||||
Raises:
|
||||
Exception: If the zip file does not exists an exception raise.
|
||||
"""
|
||||
|
||||
self.session = session
|
||||
self.archive = BACKUP_DIR.joinpath(zip_archive)
|
||||
self.imp_recipes = import_recipes
|
||||
self.imp_settings = import_settings
|
||||
@@ -75,10 +77,11 @@ class ImportDatabase:
|
||||
recipe_dict = ImportDatabase._recipe_migration(recipe_dict)
|
||||
try:
|
||||
recipe_obj = Recipe(**recipe_dict)
|
||||
recipe_obj.save_to_db()
|
||||
recipe_obj.save_to_db(self.session)
|
||||
successful_imports.append(recipe.stem)
|
||||
logger.info(f"Imported: {recipe.stem}")
|
||||
except:
|
||||
except Exception as inst:
|
||||
logger.error(inst)
|
||||
logger.info(f"Failed Import: {recipe.stem}")
|
||||
failed_imports.append(recipe.stem)
|
||||
|
||||
@@ -114,7 +117,7 @@ class ImportDatabase:
|
||||
for theme in themes:
|
||||
new_theme = SiteTheme(**theme)
|
||||
try:
|
||||
new_theme.save_to_db()
|
||||
new_theme.save_to_db(self.session)
|
||||
except:
|
||||
logger.info(f"Unable Import Theme {new_theme.name}")
|
||||
|
||||
@@ -126,7 +129,7 @@ class ImportDatabase:
|
||||
|
||||
settings = SiteSettings(**settings)
|
||||
|
||||
settings.update()
|
||||
settings.update(self.session)
|
||||
|
||||
def clean_up(self):
|
||||
shutil.rmtree(TEMP_DIR)
|
||||
|
||||
@@ -2,9 +2,7 @@ import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
|
||||
CWD = Path(__file__).parent
|
||||
IMG_DIR = CWD.parent.joinpath("data", "img")
|
||||
from app_config import IMG_DIR
|
||||
|
||||
|
||||
def read_image(recipe_slug: str) -> Path:
|
||||
|
||||
@@ -4,6 +4,7 @@ from typing import List, Optional
|
||||
|
||||
from db.database import db
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy.orm.session import Session
|
||||
|
||||
from services.recipe_services import Recipe
|
||||
|
||||
@@ -31,6 +32,7 @@ class Meal(BaseModel):
|
||||
|
||||
|
||||
class MealData(BaseModel):
|
||||
name: Optional[str]
|
||||
slug: str
|
||||
dateText: str
|
||||
|
||||
@@ -53,12 +55,12 @@ class MealPlan(BaseModel):
|
||||
}
|
||||
}
|
||||
|
||||
def process_meals(self):
|
||||
def process_meals(self, session: Session):
|
||||
meals = []
|
||||
for x, meal in enumerate(self.meals):
|
||||
|
||||
try:
|
||||
recipe = Recipe.get_by_slug(meal.slug)
|
||||
recipe = Recipe.get_by_slug(session, meal.slug)
|
||||
|
||||
meal_data = {
|
||||
"slug": recipe.slug,
|
||||
@@ -78,27 +80,29 @@ class MealPlan(BaseModel):
|
||||
|
||||
self.meals = meals
|
||||
|
||||
def save_to_db(self):
|
||||
db.meals.save_new(self.dict())
|
||||
def save_to_db(self, session: Session):
|
||||
db.meals.save_new(session, self.dict())
|
||||
|
||||
@staticmethod
|
||||
def get_all() -> List:
|
||||
def get_all(session: Session) -> List:
|
||||
|
||||
all_meals = [MealPlan(**x) for x in db.meals.get_all(order_by="startDate")]
|
||||
all_meals = [
|
||||
MealPlan(**x) for x in db.meals.get_all(session, order_by="startDate")
|
||||
]
|
||||
|
||||
return all_meals
|
||||
|
||||
def update(self, uid):
|
||||
db.meals.update(uid, self.dict())
|
||||
def update(self, session, uid):
|
||||
db.meals.update(session, uid, self.dict())
|
||||
|
||||
@staticmethod
|
||||
def delete(uid):
|
||||
db.meals.delete(uid)
|
||||
def delete(session, uid):
|
||||
db.meals.delete(session, uid)
|
||||
|
||||
@staticmethod
|
||||
def today() -> str:
|
||||
def today(session: Session) -> str:
|
||||
""" Returns the meal slug for Today """
|
||||
meal_plan = db.meals.get_all(limit=1, order_by="startDate")
|
||||
meal_plan = db.meals.get_all(session, limit=1, order_by="startDate")
|
||||
|
||||
meal_docs = [Meal(**meal) for meal in meal_plan["meals"]]
|
||||
|
||||
@@ -109,7 +113,7 @@ class MealPlan(BaseModel):
|
||||
return "No Meal Today"
|
||||
|
||||
@staticmethod
|
||||
def this_week():
|
||||
meal_plan = db.meals.get_all(limit=1, order_by="startDate")
|
||||
def this_week(session: Session):
|
||||
meal_plan = db.meals.get_all(session, limit=1, order_by="startDate")
|
||||
|
||||
return meal_plan
|
||||
|
||||
@@ -1,32 +1,17 @@
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import git
|
||||
import yaml
|
||||
from app_config import IMG_DIR, TEMP_DIR
|
||||
from services.recipe_services import Recipe
|
||||
from app_config import IMG_DIR
|
||||
from sqlalchemy.orm.session import Session
|
||||
from utils.unzip import unpack_zip
|
||||
|
||||
try:
|
||||
from yaml import CLoader as Loader
|
||||
except ImportError:
|
||||
from yaml import Loader
|
||||
|
||||
TEMP_DIR = Path(__file__).parent.parent.parent.joinpath("temp")
|
||||
|
||||
|
||||
def pull_repo(repo):
|
||||
dest_dir = TEMP_DIR.joinpath("/migration/git_pull")
|
||||
if dest_dir.exists():
|
||||
shutil.rmtree(dest_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
git.Git(dest_dir).clone(repo)
|
||||
|
||||
repo_name = repo.split("/")[-1]
|
||||
recipe_dir = dest_dir.joinpath(repo_name, "_recipes")
|
||||
image_dir = dest_dir.joinpath(repo_name, "images")
|
||||
|
||||
return recipe_dir, image_dir
|
||||
|
||||
|
||||
def read_chowdown_file(recipe_file: Path) -> Recipe:
|
||||
"""Parse through the yaml file to try and pull out the relavent information.
|
||||
@@ -75,25 +60,31 @@ def read_chowdown_file(recipe_file: Path) -> Recipe:
|
||||
return new_recipe
|
||||
|
||||
|
||||
def chowdown_migrate(repo):
|
||||
recipe_dir, image_dir = pull_repo(repo)
|
||||
def chowdown_migrate(session: Session, zip_file: Path):
|
||||
temp_dir = unpack_zip(zip_file)
|
||||
|
||||
failed_images = []
|
||||
for image in image_dir.iterdir():
|
||||
try:
|
||||
shutil.copy(image, IMG_DIR.joinpath(image.name))
|
||||
except:
|
||||
failed_images.append(image.name)
|
||||
with temp_dir as dir:
|
||||
image_dir = TEMP_DIR.joinpath(dir, zip_file.stem, "images")
|
||||
recipe_dir = TEMP_DIR.joinpath(dir, zip_file.stem, "_recipes")
|
||||
|
||||
failed_recipes = []
|
||||
for recipe in recipe_dir.glob("*.md"):
|
||||
try:
|
||||
new_recipe = read_chowdown_file(recipe)
|
||||
new_recipe.save_to_db()
|
||||
failed_recipes = []
|
||||
successful_recipes = []
|
||||
for recipe in recipe_dir.glob("*.md"):
|
||||
try:
|
||||
new_recipe = read_chowdown_file(recipe)
|
||||
new_recipe.save_to_db(session)
|
||||
successful_recipes.append(recipe.stem)
|
||||
except:
|
||||
failed_recipes.append(recipe.stem)
|
||||
|
||||
except:
|
||||
failed_recipes.append(recipe.name)
|
||||
failed_images = []
|
||||
for image in image_dir.iterdir():
|
||||
try:
|
||||
if not image.stem in failed_recipes:
|
||||
shutil.copy(image, IMG_DIR.joinpath(image.name))
|
||||
except:
|
||||
failed_images.append(image.name)
|
||||
|
||||
report = {"failedImages": failed_images, "failedRecipes": failed_recipes}
|
||||
report = {"successful": successful_recipes, "failed": failed_recipes}
|
||||
|
||||
return report
|
||||
|
||||
@@ -4,13 +4,11 @@ import shutil
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
from app_config import IMG_DIR, MIGRATION_DIR, TEMP_DIR
|
||||
from services.recipe_services import Recipe
|
||||
from services.scrape_services import normalize_data, process_recipe_data
|
||||
from app_config import IMG_DIR, TEMP_DIR
|
||||
|
||||
CWD = Path(__file__).parent
|
||||
MIGRTAION_DIR = CWD.parent.parent.joinpath("data", "migration")
|
||||
|
||||
|
||||
def process_selection(selection: Path) -> Path:
|
||||
if selection.is_dir():
|
||||
@@ -65,10 +63,10 @@ def cleanup():
|
||||
shutil.rmtree(TEMP_DIR)
|
||||
|
||||
|
||||
def migrate(selection: str):
|
||||
def migrate(session, selection: str):
|
||||
prep()
|
||||
MIGRTAION_DIR.mkdir(exist_ok=True)
|
||||
selection = MIGRTAION_DIR.joinpath(selection)
|
||||
MIGRATION_DIR.mkdir(exist_ok=True)
|
||||
selection = MIGRATION_DIR.joinpath(selection)
|
||||
|
||||
nextcloud_dir = process_selection(selection)
|
||||
|
||||
@@ -76,9 +74,10 @@ def migrate(selection: str):
|
||||
failed_imports = []
|
||||
for dir in nextcloud_dir.iterdir():
|
||||
if dir.is_dir():
|
||||
|
||||
try:
|
||||
recipe = import_recipes(dir)
|
||||
recipe.save_to_db()
|
||||
recipe.save_to_db(session)
|
||||
successful_imports.append(recipe.name)
|
||||
except:
|
||||
logging.error(f"Failed Nextcloud Import: {dir.name}")
|
||||
|
||||
@@ -6,6 +6,7 @@ from typing import Any, List, Optional
|
||||
from db.database import db
|
||||
from pydantic import BaseModel, validator
|
||||
from slugify import slugify
|
||||
from sqlalchemy.orm.session import Session
|
||||
|
||||
from services.image_services import delete_image
|
||||
|
||||
@@ -34,14 +35,13 @@ class Recipe(BaseModel):
|
||||
|
||||
# Mealie Specific
|
||||
slug: Optional[str] = ""
|
||||
categories: Optional[List[str]]
|
||||
tags: Optional[List[str]]
|
||||
categories: Optional[List[str]] = []
|
||||
tags: Optional[List[str]] = []
|
||||
dateAdded: Optional[datetime.date]
|
||||
notes: Optional[List[RecipeNote]]
|
||||
rating: Optional[int]
|
||||
notes: Optional[List[RecipeNote]] = []
|
||||
rating: Optional[int]
|
||||
orgURL: Optional[str]
|
||||
extras: Optional[dict]
|
||||
extras: Optional[dict] = {}
|
||||
|
||||
class Config:
|
||||
schema_extra = {
|
||||
@@ -82,23 +82,14 @@ class Recipe(BaseModel):
|
||||
return slug
|
||||
|
||||
@classmethod
|
||||
def _unpack_doc(cls, document):
|
||||
document = json.loads(document.to_json())
|
||||
del document["_id"]
|
||||
|
||||
document["dateAdded"] = document["dateAdded"]["$date"]
|
||||
|
||||
return cls(**document)
|
||||
|
||||
@classmethod
|
||||
def get_by_slug(cls, slug: str):
|
||||
def get_by_slug(cls, session, slug: str):
|
||||
""" Returns a Recipe Object by Slug """
|
||||
|
||||
document = db.recipes.get(slug, "slug")
|
||||
document = db.recipes.get(session, slug, "slug")
|
||||
|
||||
return cls(**document)
|
||||
|
||||
def save_to_db(self) -> str:
|
||||
def save_to_db(self, session) -> str:
|
||||
recipe_dict = self.dict()
|
||||
|
||||
try:
|
||||
@@ -113,55 +104,37 @@ class Recipe(BaseModel):
|
||||
# except:
|
||||
# pass
|
||||
|
||||
recipe_doc = db.recipes.save_new(recipe_dict)
|
||||
recipe_doc = db.recipes.save_new(session, recipe_dict)
|
||||
recipe = Recipe(**recipe_doc)
|
||||
|
||||
return recipe.slug
|
||||
|
||||
@staticmethod
|
||||
def delete(recipe_slug: str) -> str:
|
||||
def delete(session: Session, recipe_slug: str) -> str:
|
||||
""" Removes the recipe from the database by slug """
|
||||
delete_image(recipe_slug)
|
||||
db.recipes.delete(recipe_slug)
|
||||
db.recipes.delete(session, recipe_slug)
|
||||
return "Document Deleted"
|
||||
|
||||
def update(self, recipe_slug: str):
|
||||
def update(self, session: Session, recipe_slug: str):
|
||||
""" Updates the recipe from the database by slug"""
|
||||
updated_slug = db.recipes.update(recipe_slug, self.dict())
|
||||
updated_slug = db.recipes.update(session, recipe_slug, self.dict())
|
||||
return updated_slug.get("slug")
|
||||
|
||||
@staticmethod
|
||||
def update_image(slug: str, extension: str):
|
||||
db.recipes.update_image(slug, extension)
|
||||
def update_image(slug: str, extension: str) -> str:
|
||||
"""A helper function to pass the new image name and extension
|
||||
into the database.
|
||||
|
||||
Args:
|
||||
slug (str): The current recipe slug
|
||||
extension (str): the file extension of the new image
|
||||
"""
|
||||
return db.recipes.update_image(slug, extension)
|
||||
|
||||
@staticmethod
|
||||
def get_all():
|
||||
return db.recipes.get_all()
|
||||
def get_all(session: Session):
|
||||
return db.recipes.get_all(session)
|
||||
|
||||
|
||||
def read_requested_values(keys: list, max_results: int = 0) -> List[dict]:
|
||||
"""
|
||||
Pass in a list of key values to be run against the database. If a match is found
|
||||
it is then added to a dictionary inside of a list. If a key does not exist the
|
||||
it will simply not be added to the return data.
|
||||
|
||||
Parameters:
|
||||
keys: list
|
||||
|
||||
Returns: returns a list of dicts containing recipe data
|
||||
|
||||
"""
|
||||
recipe_list = []
|
||||
for recipe in db.recipes.get_all(limit=max_results, order_by="dateAdded"):
|
||||
recipe_details = {}
|
||||
for key in keys:
|
||||
try:
|
||||
recipe_key = {key: recipe[key]}
|
||||
except:
|
||||
continue
|
||||
|
||||
recipe_details.update(recipe_key)
|
||||
|
||||
recipe_list.append(recipe_details)
|
||||
|
||||
return recipe_list
|
||||
|
||||
3
mealie/services/scheduler/global_scheduler.py
Normal file
3
mealie/services/scheduler/global_scheduler.py
Normal file
@@ -0,0 +1,3 @@
|
||||
from apscheduler.schedulers.background import BackgroundScheduler
|
||||
|
||||
scheduler = BackgroundScheduler()
|
||||
62
mealie/services/scheduler/scheduled_jobs.py
Normal file
62
mealie/services/scheduler/scheduled_jobs.py
Normal file
@@ -0,0 +1,62 @@
|
||||
from apscheduler.schedulers.background import BackgroundScheduler
|
||||
from db.db_setup import create_session
|
||||
from services.backups.exports import auto_backup_job
|
||||
from services.scheduler.global_scheduler import scheduler
|
||||
from services.scheduler.scheduler_utils import Cron, cron_parser
|
||||
from services.settings_services import SiteSettings
|
||||
from utils.logger import logger
|
||||
from utils.post_webhooks import post_webhooks
|
||||
|
||||
|
||||
@scheduler.scheduled_job(trigger="interval", minutes=15)
|
||||
def update_webhook_schedule():
|
||||
"""
|
||||
A scheduled background job that runs every 15 minutes to
|
||||
poll the database for changes and reschedule the webhook time
|
||||
"""
|
||||
session = create_session()
|
||||
settings = SiteSettings.get_site_settings(session=session)
|
||||
time = cron_parser(settings.webhooks.webhookTime)
|
||||
job = JOB_STORE.get("webhooks")
|
||||
|
||||
scheduler.reschedule_job(
|
||||
job.scheduled_task.id,
|
||||
trigger="cron",
|
||||
hour=time.hours,
|
||||
minute=time.minutes,
|
||||
)
|
||||
|
||||
session.close()
|
||||
logger.info(scheduler.print_jobs())
|
||||
|
||||
|
||||
class ScheduledFunction:
|
||||
def __init__(
|
||||
self, scheduler: BackgroundScheduler, function, cron: Cron, name: str
|
||||
) -> None:
|
||||
self.scheduled_task = scheduler.add_job(
|
||||
function,
|
||||
trigger="cron",
|
||||
name=name,
|
||||
hour=cron.hours,
|
||||
minute=cron.minutes,
|
||||
max_instances=1,
|
||||
replace_existing=True,
|
||||
)
|
||||
|
||||
logger.info("New Function Scheduled")
|
||||
logger.info(scheduler.print_jobs())
|
||||
|
||||
|
||||
logger.info("----INIT SCHEDULE OBJECT-----")
|
||||
|
||||
JOB_STORE = {
|
||||
"backup_job": ScheduledFunction(
|
||||
scheduler, auto_backup_job, Cron(hours=00, minutes=00), "backups"
|
||||
),
|
||||
"webhooks": ScheduledFunction(
|
||||
scheduler, post_webhooks, Cron(hours=00, minutes=00), "webhooks"
|
||||
),
|
||||
}
|
||||
|
||||
scheduler.start()
|
||||
10
mealie/services/scheduler/scheduler_utils.py
Normal file
10
mealie/services/scheduler/scheduler_utils.py
Normal file
@@ -0,0 +1,10 @@
|
||||
import collections
|
||||
|
||||
Cron = collections.namedtuple("Cron", "hours minutes")
|
||||
|
||||
|
||||
def cron_parser(time_str: str) -> Cron:
|
||||
time = time_str.split(":")
|
||||
cron = Cron(hours=int(time[0]), minutes=int(time[1]))
|
||||
|
||||
return cron
|
||||
@@ -1,72 +0,0 @@
|
||||
import collections
|
||||
import json
|
||||
|
||||
import requests
|
||||
from apscheduler.schedulers.background import BackgroundScheduler
|
||||
from utils.logger import logger
|
||||
|
||||
from services.backups.exports import auto_backup_job
|
||||
from services.meal_services import MealPlan
|
||||
from services.recipe_services import Recipe
|
||||
from services.settings_services import SiteSettings
|
||||
|
||||
Cron = collections.namedtuple("Cron", "hours minutes")
|
||||
|
||||
|
||||
def cron_parser(time_str: str) -> Cron:
|
||||
time = time_str.split(":")
|
||||
cron = Cron(hours=int(time[0]), minutes=int(time[1]))
|
||||
|
||||
return cron
|
||||
|
||||
|
||||
def post_webhooks():
|
||||
all_settings = SiteSettings.get_site_settings()
|
||||
|
||||
if all_settings.webhooks.enabled:
|
||||
todays_meal = Recipe.get_by_slug(MealPlan.today()).dict()
|
||||
urls = all_settings.webhooks.webhookURLs
|
||||
|
||||
for url in urls:
|
||||
requests.post(url, json.dumps(todays_meal, default=str))
|
||||
|
||||
|
||||
class Scheduler:
|
||||
def startup_scheduler(self):
|
||||
self.scheduler = BackgroundScheduler()
|
||||
logger.info("----INIT SCHEDULE OBJECT-----")
|
||||
self.scheduler.start()
|
||||
|
||||
self.scheduler.add_job(
|
||||
auto_backup_job, trigger="cron", hour="3", max_instances=1
|
||||
)
|
||||
settings = SiteSettings.get_site_settings()
|
||||
time = cron_parser(settings.webhooks.webhookTime)
|
||||
|
||||
self.webhook = self.scheduler.add_job(
|
||||
post_webhooks,
|
||||
trigger="cron",
|
||||
name="webhooks",
|
||||
hour=time.hours,
|
||||
minute=time.minutes,
|
||||
max_instances=1,
|
||||
)
|
||||
|
||||
logger.info(self.scheduler.print_jobs())
|
||||
|
||||
def reschedule_webhooks(self):
|
||||
"""
|
||||
Reads the site settings database entry to reschedule the webhooks task
|
||||
Called after each post to the webhooks endpoint.
|
||||
"""
|
||||
settings = SiteSettings.get_site_settings()
|
||||
time = cron_parser(settings.webhooks.webhookTime)
|
||||
|
||||
self.scheduler.reschedule_job(
|
||||
self.webhook.id,
|
||||
trigger="cron",
|
||||
hour=time.hours,
|
||||
minute=time.minutes,
|
||||
)
|
||||
|
||||
logger.info(self.scheduler.print_jobs())
|
||||
@@ -1,10 +1,12 @@
|
||||
import html
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
from typing import List, Tuple
|
||||
|
||||
import extruct
|
||||
import requests
|
||||
import scrape_schema_recipe
|
||||
from app_config import DEBUG_DIR
|
||||
from slugify import slugify
|
||||
from utils.logger import logger
|
||||
from w3lib.html import get_base_url
|
||||
@@ -12,8 +14,13 @@ from w3lib.html import get_base_url
|
||||
from services.image_services import scrape_image
|
||||
from services.recipe_services import Recipe
|
||||
|
||||
CWD = Path(__file__).parent
|
||||
TEMP_FILE = CWD.parent.joinpath("data", "debug", "last_recipe.json")
|
||||
TEMP_FILE = DEBUG_DIR.joinpath("last_recipe.json")
|
||||
|
||||
|
||||
def cleanhtml(raw_html):
|
||||
cleanr = re.compile("<.*?>")
|
||||
cleantext = re.sub(cleanr, "", raw_html)
|
||||
return cleantext
|
||||
|
||||
|
||||
def normalize_image_url(image) -> str:
|
||||
@@ -31,17 +38,19 @@ def normalize_instructions(instructions) -> List[dict]:
|
||||
# One long string split by (possibly multiple) new lines
|
||||
if type(instructions) == str:
|
||||
return [
|
||||
{"text": line.strip()} for line in filter(None, instructions.splitlines())
|
||||
{"text": normalize_instruction(line)}
|
||||
for line in instructions.splitlines()
|
||||
if line
|
||||
]
|
||||
|
||||
# Plain strings in a list
|
||||
elif type(instructions) == list and type(instructions[0]) == str:
|
||||
return [{"text": step.strip()} for step in instructions]
|
||||
return [{"text": normalize_instruction(step)} for step in instructions]
|
||||
|
||||
# Dictionaries (let's assume it's a HowToStep) in a list
|
||||
elif type(instructions) == list and type(instructions[0]) == dict:
|
||||
return [
|
||||
{"text": step["text"].strip()}
|
||||
{"text": normalize_instruction(step["text"])}
|
||||
for step in instructions
|
||||
if step["@type"] == "HowToStep"
|
||||
]
|
||||
@@ -50,6 +59,19 @@ def normalize_instructions(instructions) -> List[dict]:
|
||||
raise Exception(f"Unrecognised instruction format: {instructions}")
|
||||
|
||||
|
||||
def normalize_instruction(line) -> str:
|
||||
l = cleanhtml(line.strip())
|
||||
# Some sites erroneously escape their strings on multiple levels
|
||||
while not l == (l := html.unescape(l)):
|
||||
pass
|
||||
return l
|
||||
|
||||
|
||||
def normalize_ingredient(ingredients: list) -> str:
|
||||
|
||||
return [cleanhtml(html.unescape(ing)) for ing in ingredients]
|
||||
|
||||
|
||||
def normalize_yield(yld) -> str:
|
||||
if type(yld) == list:
|
||||
return yld[-1]
|
||||
@@ -66,9 +88,13 @@ def normalize_time(time_entry) -> str:
|
||||
|
||||
def normalize_data(recipe_data: dict) -> dict:
|
||||
recipe_data["totalTime"] = normalize_time(recipe_data.get("totalTime"))
|
||||
recipe_data["description"] = cleanhtml(recipe_data.get("description", ""))
|
||||
recipe_data["prepTime"] = normalize_time(recipe_data.get("prepTime"))
|
||||
recipe_data["performTime"] = normalize_time(recipe_data.get("performTime"))
|
||||
recipe_data["recipeYield"] = normalize_yield(recipe_data.get("recipeYield"))
|
||||
recipe_data["recipeIngredient"] = normalize_ingredient(
|
||||
recipe_data.get("recipeIngredient")
|
||||
)
|
||||
recipe_data["recipeInstructions"] = normalize_instructions(
|
||||
recipe_data["recipeInstructions"]
|
||||
)
|
||||
@@ -165,7 +191,7 @@ def process_recipe_url(url: str) -> dict:
|
||||
return new_recipe
|
||||
|
||||
|
||||
def create_from_url(url: str) -> dict:
|
||||
def create_from_url(url: str) -> Recipe:
|
||||
recipe_data = process_recipe_url(url)
|
||||
|
||||
with open(TEMP_FILE, "w") as f:
|
||||
@@ -173,4 +199,4 @@ def create_from_url(url: str) -> dict:
|
||||
|
||||
recipe = Recipe(**recipe_data)
|
||||
|
||||
return recipe.save_to_db()
|
||||
return recipe
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
from typing import List, Optional
|
||||
|
||||
from db.database import db
|
||||
from db.db_setup import sql_exists
|
||||
from db.db_setup import create_session, sql_exists
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy.orm.session import Session
|
||||
from utils.logger import logger
|
||||
|
||||
|
||||
@@ -29,22 +30,24 @@ class SiteSettings(BaseModel):
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def get_all():
|
||||
db.settings.get_all()
|
||||
def get_all(session: Session):
|
||||
db.settings.get_all(session)
|
||||
|
||||
@classmethod
|
||||
def get_site_settings(cls):
|
||||
def get_site_settings(cls, session: Session):
|
||||
try:
|
||||
document = db.settings.get("main")
|
||||
document = db.settings.get(session=session, match_value="main")
|
||||
except:
|
||||
webhooks = Webhooks()
|
||||
default_entry = SiteSettings(name="main", webhooks=webhooks)
|
||||
document = db.settings.save_new(default_entry.dict(), webhooks.dict())
|
||||
document = db.settings.save_new(
|
||||
session, default_entry.dict(), webhooks.dict()
|
||||
)
|
||||
|
||||
return cls(**document)
|
||||
|
||||
def update(self):
|
||||
db.settings.update("main", new_data=self.dict())
|
||||
def update(self, session: Session):
|
||||
db.settings.update(session, "main", new_data=self.dict())
|
||||
|
||||
|
||||
class Colors(BaseModel):
|
||||
@@ -78,16 +81,16 @@ class SiteTheme(BaseModel):
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def get_by_name(cls, theme_name):
|
||||
db_entry = db.themes.get(theme_name)
|
||||
def get_by_name(cls, session: Session, theme_name):
|
||||
db_entry = db.themes.get(session, theme_name)
|
||||
name = db_entry.get("name")
|
||||
colors = Colors(**db_entry.get("colors"))
|
||||
|
||||
return cls(name=name, colors=colors)
|
||||
|
||||
@staticmethod
|
||||
def get_all():
|
||||
all_themes = db.themes.get_all()
|
||||
def get_all(session: Session):
|
||||
all_themes = db.themes.get_all(session)
|
||||
for index, theme in enumerate(all_themes):
|
||||
name = theme.get("name")
|
||||
colors = Colors(**theme.get("colors"))
|
||||
@@ -96,16 +99,16 @@ class SiteTheme(BaseModel):
|
||||
|
||||
return all_themes
|
||||
|
||||
def save_to_db(self):
|
||||
db.themes.save_new(self.dict())
|
||||
def save_to_db(self, session: Session):
|
||||
db.themes.save_new(session, self.dict())
|
||||
|
||||
def update_document(self):
|
||||
db.themes.update(self.dict())
|
||||
def update_document(self, session: Session):
|
||||
db.themes.update(session, self.name, self.dict())
|
||||
|
||||
@staticmethod
|
||||
def delete_theme(theme_name: str) -> str:
|
||||
def delete_theme(session: Session, theme_name: str) -> str:
|
||||
""" Removes the theme by name """
|
||||
db.themes.delete(theme_name)
|
||||
db.themes.delete(session, theme_name)
|
||||
|
||||
|
||||
def default_theme_init():
|
||||
@@ -118,24 +121,27 @@ def default_theme_init():
|
||||
"warning": "#FF4081",
|
||||
"error": "#EF5350",
|
||||
}
|
||||
|
||||
session = create_session()
|
||||
try:
|
||||
SiteTheme.get_by_name("default")
|
||||
SiteTheme.get_by_name(session, "default")
|
||||
logger.info("Default theme exists... skipping generation")
|
||||
except:
|
||||
logger.info("Generating Default Theme")
|
||||
colors = Colors(**default_colors)
|
||||
default_theme = SiteTheme(name="default", colors=colors)
|
||||
default_theme.save_to_db()
|
||||
default_theme.save_to_db(session)
|
||||
|
||||
|
||||
def default_settings_init():
|
||||
session = create_session()
|
||||
try:
|
||||
document = db.settings.get("main")
|
||||
document = db.settings.get(session, "main")
|
||||
except:
|
||||
webhooks = Webhooks()
|
||||
default_entry = SiteSettings(name="main", webhooks=webhooks)
|
||||
document = db.settings.save_new(default_entry.dict(), webhooks.dict())
|
||||
document = db.settings.save_new(session, default_entry.dict(), webhooks.dict())
|
||||
|
||||
session.close()
|
||||
|
||||
|
||||
if not sql_exists:
|
||||
|
||||
Reference in New Issue
Block a user