Product Opener/Installation/Debian or Ubuntu: Difference between revisions

VisualWikitext

Revision as of 12:02, 11 February 2015

Also see Product Opener Installation

This page lists all the steps necessary to install Product Opener on Debian, including a development environment.

All instructions below are for user "stephane", please use your own name. :-)

Open Food Facts test / development server

OFF test / development server is a small dedicated server ("kimsufi 2G", the cheap sub-brand of OVH).

Linux ks3095298.kimsufi.com 3.10.23-xxxx-std-ipv6-64 #1 SMP Tue Mar 18 14:48:24 CET 2014 x86_64 GNU/Linux
CPU: Intel(R) Atom(TM) CPU 230 @ 1.60GHz - Cores: 2 - Cache: 512KB
RAM: 2 Gb
Disk: 1 Tb

Debian install

Fresh Debian install with OVH / kimsufi install process:

Step 1:
- OS - Type: Basic
- Debian 7.5 stable (Wheezy) - debian 7 - 64 bits
Step 3: Partitions
- Default:
  - / ext4 20 Gb
  - /home/ ext4 979.5 Gb
  - swap 1 x 512 Mb

DNS

ks3095298.kimsufi.com IP address: 94.23.195.82

Create 2 A records in the DNS zone of the domain you want to use:

openfoodfacts.ovh 94.23.195.82
*.openfoodfacts.ovh 94.23.195.82

Install

Update Debian

Usually done by OVH Debian install:

apt-get update
apt-get upgrade

Security

Install fail2ban and sudo:

apt-get install fail2ban
apt-get install sudo

Create an user and give it sudo access:

adduser stephane
adduser stephane sudo

Install software used by Product Opener

apt-get install mongodb

apt-get install memcached

apt-get install imagemagick

apt-get install tesseract-ocr
apt-get install tesseract-ocr-fra

apt-get install exim4
dpkg-reconfigure exim4-config
- Internet Site mail is sent by smtp  
- 127.0.0.1

apt-get install geoip-bin geoip-database

Install libraries used by Product Opener

apt-get install zlib1g-dev

Install development tools

Note: the Product Opener code is currently on a private bitbucket.org repository and is in the process of being made publicly available (open source). If you want to help us to clean / document Product Opener to open it, request access to the bitbucket repository: stephane@openfoodfacts.org

apt-get install build-essential
apt-get install git

git clone https://stephane@bitbucket.org/openfoodfacts/product-opener.git

(get the link for your user on bitbucket.org)

Apache 2.2 + mod_perl: Product Opener backend

Build

apt-get install libperl-dev

cd /home/stephane
mkdir src
cd src
wget http://apache.crihan.fr/dist//httpd/httpd-2.2.29.tar.gz
tar xvfz httpd-2.2.29.tar.gz
cd httpd-2.2.29
./configure --with-mpm=prefork --prefix=/home/stephane/apache  --enable-rewrite --enable-proxy --enable-proxy_http --enable-deflate  --disable-userdir --enable-headers
make
make install
cd ~

cd ~/src
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.8.tar.gz 
tar xvfz mod_perl-2.0.8.tar.gz
cd mod_perl-2.0.8
perl Makefile.PL MP_APXS=/home/stephane/apache/bin/apxs 
make
make install
cd ~

Configuration

In ~/apache/conf/httpd.conf:

Change the port to something else than 80 (the reverse proxy will listen on 80 and forward to the backend on another port).

Listen 8001

If you are installing several Product Openers on the same machine, use different ports.

Change the Apache user to your user:

User stephane
Group stephane

Add to apache/conf/httpd.conf:

LoadModule perl_module modules/mod_perl.so

PerlWarn Off
PerlRequire /home/stephane/product-opener/cgi/startup.pl


<Location /cgi>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
Options +ExecCGI
Order allow,deny
Allow from all
</Location>


<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName openfoodfacts.ovh
ErrorLog /home/stephane/logs/error_log
CustomLog /home/stephane/logs/access_log combined
ScriptAlias /cgi/ "/home/stephane/product-opener/cgi/"
</VirtualHost>

PerlPostReadRequestHandler My::ProxyRemoteAddr

Apache 2.4 - light weight reverse proxy

Build

apt-get install libpcre3 libpcre3-dev

cd /home/stephane/src
wget http://apache.websitebeheerjd.nl//httpd/httpd-2.4.12.tar.gz
tar xvfz httpd-2.4.12.tar.gz
cd httpd-2.4.12

cd srclib
wget http://mir2.ovh.net/ftp.apache.org/dist//apr/apr-1.5.1.tar.gz
wget http://mir2.ovh.net/ftp.apache.org/dist//apr/apr-util-1.5.4.tar.gz   
tar xvfz apr-1.5.1.tar.gz
tar xvfz apr-util-1.5.4.tar.gz 
ln -s apr-1.5.1 ap
ln -s apr-util-1.5.4 apr-util
cd .. 

./configure --prefix=/home/stephane/proxy --enable-rewrite --enable-proxy --enable-proxy_http --disable-userdir --enable-headers --enable-expires --enable-mime --enable-so --with-included-apr

make
make install

Configuration

in proxy/conf/httpd.conf :

uncomment #LoadModule rewrite_module modules/mod_rewrite.so
uncomment #LoadModule slotmem_shm_module modules/mod_slotmem_shm.so

Add in httpd.conf :

<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName world.openfoodfacts.ovh
ServerAlias *.openfoodfacts.ovh

ErrorLog /home/stephane/logs/proxy_error_log
CustomLog /home/stephane/logs/proxy_access_log combined
ServerAdmin stephane@openfoodfacts.org

<Directory "/home/stephane/product-opener/html">
    Options Indexes FollowSymLinks
    Require all granted
</Directory>
ProxyPreserveHost On
RewriteEngine on
RewriteCond  %{REQUEST_URI}  !/./
RewriteRule ^(/cgi/.*)$ http://localhost:8001$1 [P,L]
RewriteMap escape int:escape
RewriteRule ^/favicon.ico$ /favicon.ico [L]
RewriteCond  %{REQUEST_URI}  !^/images/
RewriteCond  %{REQUEST_URI}  !^/js/
RewriteCond  %{REQUEST_URI}  !^/rss/
RewriteCond  %{REQUEST_URI}  !^/robots
RewriteCond  %{REQUEST_URI}  !^/clicks/
RewriteCond  %{REQUEST_URI}  !^/data/
RewriteCond  %{REQUEST_URI}  !^/files/
RewriteRule  ^(.*)$ http://localhost:8001/cgi/display.pl?${escape:$1} [P,L,QSA]
</VirtualHost>


<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName openfoodfacts.ovh
ErrorLog /home/stephane/logs/proxy_error_log
CustomLog /home/stephane/logs/proxy_access_log combined
DirectoryIndex index.html index.shtml
<Directory "/home/stephane/product-opener/html">
    Options -Indexes +FollowSymLinks +Includes
    Require all granted
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} ^stephane\.openfoodfacts\.ovh
RewriteRule ^/products$ /products.shtml [L]

</VirtualHost>

Install Perl modules

If possible, use apt-get to install packages directly. If they are not available, you can use cpan to install the modules.

apt-get install libwww-perl libimage-magick-perl 
apt-get install  libxml-encoding-perl libtext-unaccent-perl libmime-lite-perl
apt-get install libcache-memcached-fast-perl libjson-perl libclone-perl  libgraphviz-perl 
apt-get install libmime-lite-perl  # needs to be done after installing exim4
apt-get install libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl 
apt-get install libxml-feedpp-perl libmongodb-perl liburi-find-perl

Modules that could not be installed using apt-get:

cpan
install URI::Escape::XS
install Image::OCR::Tesseract # needs imagemagick installed first

Modules that HAVE to be installed using cpan:

As of Feb 3rd 2015, the MongoDB perl module installed through "apt-get install libmongodb-perl" is too old (0.45):

perl -MMongoDB -e 'print $MongoDB::VERSION . "\n"'
0.45

This causes an error when trying to access a list of "tags" such as categories:

Software error:
Not an ARRAY reference at /home/stephane/product-opener/cgi/Blogs/Display.pm line 797.

A newer MongoDB perl module needs to be installed with cpan.

cpan
install MongoDB

perl -MMongoDB -e 'print $MongoDB::VERSION . "\n"'
v0.708.0.0

cgi/startup.pl preloads a lot of modules, you can check that those modules are installed by running:

cd /home/stephane/product-opener/cgi
perl startup.pl

Create some paths and links

cd /home/stephane/product-opener/cgi
ln -s ./ Blogs
ln -s S