feat: translate the site in french
This commit is contained in:
parent
3e98ac15b6
commit
b47b193b20
GPG key ID:
8E254EA0FFEB9B6D
81 changed files with 1327 additions and 251 deletions
172
content/blog/004-locking-buzhug/index.en.md
Normal file
172
content/blog/004-locking-buzhug/index.en.md
Normal file
|
|
@ -0,0 +1,172 @@
|
|||
---
|
||||
slug: 4-locking-buzhug
|
||||
title: Locking Buzhug
|
||||
date: "2012-02-07T00:00:00+01:00"
|
||||
categories: [Dev]
|
||||
tags:
|
||||
- Python
|
||||
- Buzhug
|
||||
summary: >
|
||||
How to implement a cross-process, system-wide lock for Buzhug
|
||||
---
|
||||
|
||||
I have recently decided to work with [Buzhug] on a project. As far as I can tell,
|
||||
it has proven efficient, fast, easy to use and to maintain. However, I ran into
|
||||
a few gotchas.
|
||||
|
||||
[Buzhug]: http://buzhug.sourceforge.net
|
||||
|
||||
## Simple Solutions Are Often The Bests
|
||||
|
||||
I came to use Buzhug for the following requirements:
|
||||
|
||||
- I needed a single table
|
||||
- I did not want to add additional dependencies to the project
|
||||
- The size of the table will average 5K entries (without having more than
|
||||
10k entries in peaks)
|
||||
|
||||
And an additional (personal) one:
|
||||
|
||||
- I did not want to bother with SQL. Really not. no way!
|
||||
|
||||
That left me one option: pure-python embedded database.
|
||||
|
||||
After having considered a few libraries, I have been seduced by the way Buzhug
|
||||
interface is close to manipulating python objects. And the benchmarks seemed
|
||||
to show that it is performant enough for this project.
|
||||
|
||||
After a quick prototyping (1 day), the choice was done.
|
||||
|
||||
Then came a few weeks of development and the first stress tests...
|
||||
|
||||
## And The Real World Came Back Fast
|
||||
|
||||
A few times a day, the application backed by this database is intensely used:
|
||||
|
||||
- It can be run up to 50 times simultaneously in separate python process
|
||||
- Each run makes a read and a write/delete operation
|
||||
|
||||
This causes a race condition on the files used to store data, and concurent
|
||||
writes corrupts database.
|
||||
|
||||
Using `buzhug.TS_Base` instead of `buzhug.Base` did not solve anything,
|
||||
as the problem is not thread, but processes. What I need is a system-wide
|
||||
cross-process lock.
|
||||
|
||||
## Here Is The Answer
|
||||
|
||||
First step was to find how to implement a cross-process, system-wide lock.
|
||||
As it only has to work on Linux, the
|
||||
[Lock class given by Chris from
|
||||
Vmfarms](http://blog.vmfarms.com/2011/03/cross-process-locking-and.html) fits
|
||||
perfectly. Here is a version slightly modified to make it a context manager :
|
||||
|
||||
```python
|
||||
import fcntl
|
||||
|
||||
class PsLock:
|
||||
"""
|
||||
Taken from:
|
||||
http://blog.vmfarms.com/2011/03/cross-process-locking-and.html
|
||||
"""
|
||||
def __init__(self, filename):
|
||||
self.filename = filename
|
||||
self.handle = open(filename, 'w')
|
||||
|
||||
# Bitwise OR fcntl.LOCK_NB if you need a non-blocking lock
|
||||
def acquire(self):
|
||||
fcntl.flock(self.handle, fcntl.LOCK_EX)
|
||||
|
||||
def release(self):
|
||||
fcntl.flock(self.handle, fcntl.LOCK_UN)
|
||||
|
||||
def __del__(self):
|
||||
self.handle.close()
|
||||
|
||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
||||
if exc_type is None:
|
||||
pass
|
||||
self.release()
|
||||
|
||||
def __enter__(self):
|
||||
self.acquire()
|
||||
```
|
||||
|
||||
The second step is to define a new class that inheritates from `buzhug.Base`
|
||||
that uses `PsLock` (inspired by `TS_Base`):
|
||||
|
||||
```python
|
||||
import buzhug
|
||||
|
||||
_lock = PsLock("/tmp/buzhug.lck")
|
||||
|
||||
class PS_Base(buzhug.Base):
|
||||
|
||||
def create(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.create(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def open(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.open(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def close(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.close(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def destroy(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.destroy(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def set_default(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.set_default(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def insert(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.insert(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def update(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.update(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def delete(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.delete(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def cleanup(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.cleanup(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def commit(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.commit(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def add_field(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.add_field(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def drop_field(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.drop_field(self, *args, **kw)
|
||||
return res
|
||||
```
|
||||
|
||||
Now I just use
|
||||
|
||||
```python
|
||||
database = PS_Base( ... )
|
||||
```
|
||||
|
||||
And all the errors have vanished.
|
||||
177
content/blog/004-locking-buzhug/index.fr.md
Normal file
177
content/blog/004-locking-buzhug/index.fr.md
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
---
|
||||
slug: 4-verrouiller-buzhug
|
||||
title: Verrouiller de Buzhug
|
||||
date: "2012-02-07T00:00:00+01:00"
|
||||
categories: [Dev]
|
||||
tags:
|
||||
- Python
|
||||
- Buzhug
|
||||
summary: >
|
||||
Comment implémenter un verrou inter-processus et système pour Buzhug
|
||||
---
|
||||
|
||||
J'ai récemment décidé d'utiliser [Buzhug] pour un projet. À ce que je puisse en
|
||||
juger, il s'est avéré efficace, rapide, facile à utiliser et à maintenir.
|
||||
Cependant, j'ai rencontré quelques problèmes.
|
||||
|
||||
[Buzhug]: http://buzhug.sourceforge.net
|
||||
|
||||
## Les solutions simples sont souvent les meilleures
|
||||
|
||||
J'en suis venu à utiliser Buzhug pour les raisons suivantes :
|
||||
|
||||
- J'avais besoin d'une seule table
|
||||
- Je ne voulais pas ajouter de dépendances supplémentaires au projet
|
||||
- La taille de la table serait en moyenne de 5K entrées (sans dépasser
|
||||
10k entrées en pic)
|
||||
|
||||
Et une raison supplémentaire (personnelle) :
|
||||
|
||||
- Je ne voulais pas me soucier de SQL. Vraiment pas. Pas question !
|
||||
|
||||
Cela ne me laissait qu'une option : une base de données embarquée en pur Python.
|
||||
|
||||
Après avoir considéré quelques bibliothèques, j'ai été séduit par la manière
|
||||
dont l'interface de Buzhug est proche de la manipulation d'objets Python. Et les
|
||||
benchmarks semblaient montrer qu'il est assez performant pour ce projet.
|
||||
|
||||
Après un rapide prototypage (1 jour), le choix était fait.
|
||||
|
||||
Puis vinrent quelques semaines de développement et les premiers tests de
|
||||
charge...
|
||||
|
||||
## Et la réalité est revenue rapidement
|
||||
|
||||
Plusieurs fois par jour, l'application soutenue par cette base de données est
|
||||
intensément utilisée :
|
||||
|
||||
- Elle peut être exécutée jusqu'à 50 fois simultanément dans des processus
|
||||
Python séparés
|
||||
- Chaque exécution effectue une opération de lecture et une opération
|
||||
d'écriture/suppression
|
||||
|
||||
Cela provoque une condition de course sur les fichiers utilisés pour stocker les
|
||||
données, et les écritures concurrentes corrompent la base de données.
|
||||
|
||||
L'utilisation de `buzhug.TS_Base` au lieu de `buzhug.Base` n'a rien résolu, car
|
||||
le problème n'est pas lié aux threads, mais aux processus. Ce dont j'ai besoin
|
||||
est un verrou inter-processus.
|
||||
|
||||
## Voici la solution
|
||||
|
||||
La première étape a été de trouver comment implémenter un verrou inter-processus
|
||||
et système.
|
||||
|
||||
Comme cela doit seulement fonctionner sur Linux, la
|
||||
[classe Lock donnée par Chris de
|
||||
Vmfarms](http://blog.vmfarms.com/2011/03/cross-process-locking-and.html) convient
|
||||
parfaitement. Voici une version légèrement modifiée pour en faire un gestionnaire de contexte :
|
||||
|
||||
```python
|
||||
import fcntl
|
||||
|
||||
class PsLock:
|
||||
"""
|
||||
Adapté de :
|
||||
http://blog.vmfarms.com/2011/03/cross-process-locking-and.html
|
||||
"""
|
||||
def __init__(self, filename):
|
||||
self.filename = filename
|
||||
self.handle = open(filename, 'w')
|
||||
|
||||
# Faites un OR bit à bit avec fcntl.LOCK_NB si vous avez besoin d'un verrou non bloquant
|
||||
def acquire(self):
|
||||
fcntl.flock(self.handle, fcntl.LOCK_EX)
|
||||
|
||||
def release(self):
|
||||
fcntl.flock(self.handle, fcntl.LOCK_UN)
|
||||
|
||||
def __del__(self):
|
||||
self.handle.close()
|
||||
|
||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
||||
if exc_type is None:
|
||||
pass
|
||||
self.release()
|
||||
|
||||
def __enter__(self):
|
||||
self.acquire()
|
||||
```
|
||||
|
||||
La deuxième étape consiste à définir une nouvelle classe qui hérite de
|
||||
`buzhug.Base` et qui utilise `PsLock` (inspiré de `TS_Base`) :
|
||||
|
||||
```python
|
||||
import buzhug
|
||||
|
||||
_lock = PsLock("/tmp/buzhug.lck")
|
||||
|
||||
class PS_Base(buzhug.Base):
|
||||
def create(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.create(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def open(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.open(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def close(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.close(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def destroy(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.destroy(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def set_default(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.set_default(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def insert(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.insert(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def update(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.update(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def delete(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.delete(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def cleanup(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.cleanup(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def commit(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.commit(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def add_field(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.add_field(self, *args, **kw)
|
||||
return res
|
||||
|
||||
def drop_field(self, *args, **kw):
|
||||
with _lock:
|
||||
res = buzhug.Base.drop_field(self, *args, **kw)
|
||||
return res
|
||||
```
|
||||
|
||||
Maintenant, j'utilise simplement
|
||||
|
||||
```python
|
||||
database = PS_Base( ... )
|
||||
```
|
||||
|
||||
Et toutes les erreurs ont disparu.
|
||||
Loading…
Add table
Add a link
Reference in a new issue