Jump to content

Product Opener/Installation/Debian or Ubuntu: Difference between revisions

Product Opener/Installation/Debian or Ubuntu (view source)

Revision as of 13:38, 7 August 2024

14,762 bytes added ,  7 August
no edit summary
VisualWikitext
(Created page with "Also see Product Opener Installation This page lists all the steps necessary to install Product Opener on Debian, including a development environment. All instructions b...")
 
No edit summary
 
(34 intermediate revisions by 6 users not shown)
Line 1: Line 1:
Also see [[Product Opener Installation]]
Also see [[Product Opener Installation]]
[[Category:ProductOpener]]
[[Category:Infrastructure]]
This page lists all the steps necessary to install Product Opener on Debian, including a development environment.
= Warning! Those instructions are outdated and kept for reference only =
Please see http://en.wiki.openfoodfacts.org/Infrastructure for up-to-date install instructions.
= Installation on Jessie =
We start from a clean Jessie install.
== Get the code and configure it ==
<pre>
apt-get install git
git clone <repo> /srv/openfoodfacts
mkdir /srv/openfoodfacts/users
mkdir /srv/openfoodfacts/products
chown -R www-data:www-data /srv/openfoodfacts
cd /srv/openfoodfacts/cgi
rm Blogs
ln -s . Blogs
ln -s SiteLang_off.pm SiteLang.pm
</pre>
Edit a few variables in Config2.pm:
<pre>
$server_domain = "yourdomain.com";
$www_root="/srv/openfoodfacts/html";
$data_root="/srv/openfoodfacts";
</pre>
Also in Config_off.pm:
<pre>
$domain = 'yourdomain.com';
$contact_email = 'webmaster@yourdomain.com';
$admin_email = 'webmaster@yourdomain.com';
</pre>
And in startup.pl:
<pre>
use lib "/srv/openfoodfacts/cgi/";
</pre>
== Install packaged prerequisites ==
<pre>
apt-get install apache2 build-essential geoip-bin geoip-database imagemagick libbarcode-zbar-perl libcache-memcached-fast-perl libclone-perl libcrypt-passwdmd5-perl libdatetime-format-mail-perl \
libencode-detect-perl libgeo-ip-perl libgraphics-color-perl libgraphviz-perl libhtml-defang-perl libimage-magick-perl libjson-perl libmime-lite-perl libmime-lite-perl libmongodb-perl libperl-dev \
libtest-nowarnings-perl libtext-unaccent-perl liburi-escape-xs-perl liburi-find-perl libwww-perl libxml-encoding-perl libxml-feedpp-perl memcached mongodb tesseract-ocr tesseract-ocr-fra zlib1g-dev
</pre>
== Prepare CPAN ==
If you never used CPAN before, you can automatically configure it:
<pre>
(echo y;echo o conf prerequisites_policy follow;echo o conf commit)|cpan
</pre>


This page lists all the steps necessary to install Product Opener on Debian, including a development environment.
== Install a few Perl modules from CPAN ==
 
<pre>
cpan Net::IDN::Punycode
cpan Encode::Punycode
cpan GraphViz2
cpan Image::OCR::Tesseract
cpan Algorithm::CheckDigits
cpan XML::Simple
</pre>
 
Note: on Debian Jessie, you might need to install the follwoing to be able to get cpan to install GraphViz2
<pre>
apt-get install libtest-version-perl
</pre>
 
== Rebuild Apache 2.2 and mod_perl ==
 
<pre>
cd /usr/local/src/
wget http://apache.crihan.fr/dist/httpd/httpd-2.2.31.tar.gz
tar xvfz httpd-2.2.31.tar.gz
cd httpd-2.2.31
./configure --with-mpm=prefork --prefix=/usr/local  --enable-rewrite --enable-proxy --enable-proxy_http --enable-deflate --disable-userdir --enable-headers
make
make install
 
cd /usr/local/src/
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.9.tar.gz
tar xvfz mod_perl-2.0.9.tar.gz
cd mod_perl-2.0.9
perl Makefile.PL MP_APXS=/usr/local/bin/apxs
make
make install
 
cat >> /usr/local/conf/httpd.conf <<EOF
Listen 8001
 
LoadModule perl_module modules/mod_perl.so
 
PerlWarn Off
PerlRequire /srv/openfoodfacts/cgi/startup.pl
 
<VirtualHost *:8001>
  DocumentRoot /srv/openfoodfacts/html
  ServerName yourdomain.com
  ErrorLog /var/log/apache2/error.log
  CustomLog /var/log/apache2/access.log combined
  ScriptAlias /cgi/ /srv/openfoodfacts/cgi/
 
  <Location /cgi>
    SetHandler perl-script
    PerlResponseHandler ModPerl::Registry
    PerlOptions +ParseHeaders
    Options +ExecCGI
    Order allow,deny
    Allow from all
  </Location>
</VirtualHost>
 
PerlPostReadRequestHandler My::ProxyRemoteAddr
EOF
 
sed -i -e 's/^Listen 80$/#Listen 80/' /usr/local/conf/httpd.conf
sed -i -e 's/^User.*/User www-data/' /usr/local/conf/httpd.conf
sed -i -e 's/^Group.*/Group www-data/' /usr/local/conf/httpd.conf
 
/usr/local/bin/apachectl -k start
</pre>
 
== Configure standard Apache 2 ==
 
<pre>
a2enmod proxy_http
a2enmod rewrite
 
cat >> /etc/apache2/sites-available/product-opener-proxy.conf <<EOF
<VirtualHost *>
  DocumentRoot /srv/openfoodfacts/html
  ServerName yourdomain.com
  ServerAlias *.yourdomain.com
 
  <Directory "/srv/openfoodfacts/html">
    Options Indexes FollowSymLinks
    Require all granted
  </Directory>
  ProxyPreserveHost On
  RewriteEngine on
  RewriteCond  %{REQUEST_URI}  !/./
  RewriteRule ^(/cgi/.*)$ http://localhost:8001$1 [P,L]
  RewriteMap escape int:escape
  RewriteRule ^/favicon.ico$ /favicon.ico [L]
  RewriteCond  %{REQUEST_URI}  !^/images/
  RewriteCond  %{REQUEST_URI}  !^/js/
  RewriteCond  %{REQUEST_URI}  !^/foundation/
  RewriteCond  %{REQUEST_URI}  !^/rss/
  RewriteCond  %{REQUEST_URI}  !^/robots
  RewriteCond  %{REQUEST_URI}  !^/clicks/
  RewriteCond  %{REQUEST_URI}  !^/data/
  RewriteCond  %{REQUEST_URI}  !^/files/
  RewriteRule  ^(.*)$ http://localhost:8001/cgi/display.pl?${escape:$1} [P,L,QSA]
</VirtualHost>
EOF
 
a2ensite product-opener-proxy.conf
a2dissite 000-default.conf
service apache2 restart
</pre>
 
You should now have a working Productopener instance running at http://yourdomain.com/. Enjoy!
 
= Installation on Debian Jessie with stock Apache and nginx =
 
<pre>
Vous êtes sur le point d'installer un OS sur le serveur ks3095298.kimsufi.com. Choisissez le type d'installation que vous souhaitez utiliser :
 
-> Installer à partir d'un template OVH
 
Debian 8.1 stable (Jessie) (stable) - debian8
 
root@ks3095298:~# uname -a
Linux ks3095298.kimsufi.com 3.14.32-xxxx-grs-ipv6-64 #5 SMP Wed Sep 9 17:24:34 CEST 2015 x86_64 GNU/Linux
 
perl -v
 
This is perl 5, version 20, subversion 2 (v5.20.2) built for x86_64-linux-gnu-thread-multi
 
 
apt-get update
apt-get upgrade
 
apt-get install fail2ban
apt-get install sudo
 
apt-get install mongodb
 
MongoDB shell version: 2.4.10
 
vi /etc/mongodb.conf
- change dbpath
- uncomment :
#port = 27017
#nohttpinterface = true
 
 
apt-get install git
 
apt-get install apache2
apt-get install libapache2-mod-perl2
apt-get install modperl
 
adduser off
 
apt-get install imagemagick
apt-get install memcached
apt-get install tesseract-ocr tesseract-ocr-fra
apt-get install exim4
dpkg-reconfigure exim4-config
apt-get install geoip-bin geoip-database
apt-get install zlib1g-dev
 
apt-get install libwww-perl libimage-magick-perl libxml-encoding-perl libtext-unaccent-perl libmime-lite-perl libcache-memcached-fast-perl libjson-perl libclone-perl  libgraphviz-perl libmime-lite-perl libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl libxml-feedpp-perl libmongodb-perl liburi-find-perl libxml-simple-perl
 
 
cp -a /home/openfoodfacts-bitbucket/product-opener/cgi /home/off/cgi/
 
/home/off/cgi# rm Blogs
root@ks3095298:/home/off/cgi# ln -s /home/off/cgi Blogs
 
ln -s SiteLang_off.pm SiteLang.pm
 
vi /home/off/startup.pl
# Needs to be configured
use lib "/home/off/cgi/";
 
 
ln -s ../sites-available/001-off.conf  001-off.conf
service apache2 restart
 
To see errors, missing perl modules etc.
 
/var/log/apache2# tail -f error.log
 
cpan
install URI::Escape::XS
install Encode::Punycode
install GraphViz2
install HTML::Defang
install Algorithm::CheckDigits
install Geo::IP
install Image::OCR::Tesseract
DateTime::Format::Mail
DateTime::Format::CLDR
DateTime::Locale
 
 
 
 
Issues:
   
MongoDB does not start:
   
    service mongodb start
root@ks3095298:/home/off/cgi# ps -ef | grep mongo
root    21814 23471  0 23:56 pts/0    00:00:00 grep mongo
 
 
in /var/log/syslog:
   
    Jan 11 00:05:05 ks3095298 systemd[1]: Starting An object/document-oriented database...
Jan 11 00:05:05 ks3095298 systemd[1]: Started An object/document-oriented database.
Jan 11 00:05:05 ks3095298 mongod[22404]: Mon Jan 11 00:05:05.434 terminate() called, printing stack (if implemented for platform):
Jan 11 00:05:05 ks3095298 mongod[22404]: 0xa6d3c6 0x57e204 0x7212186b7bb6 0x7212186b7c01 0x7212186b7e19 0x7212192b666d 0xa6070d 0xa61a67 0xa5f8ca 0xa5f979 0x5a626c 0x5a65d2 0x5a6a93 0x5a6ab9 0x5a6e5c 0x56647b 0x721217dbab45 0x57e05c
Jan 11 00:05:05 ks3095298 mongod[22404]: /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xa6d3c6]
 
 
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com --recv EA312927
 
 
Text::Unaccent was causing Apache to segfault, replaced the module with pure perl
 
remote_ip has been replaced in Apache 2.4, need to check if we do get the X-Forwarder-For ip directly now
 
 
Loading products from /products in the mongodb database:
   
update_all_products_from_dir_in_mongodb.pl
   
   
 
Trying to use nginx for the reverse proxy:
   
apt-get install nginx  (need port 80 to be available, stop apache or change its port first)
 
 
 
root@ks3095298:/etc/nginx/sites-enabled# more /etc/nginx/sites-available/off
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# http://wiki.nginx.org/Pitfalls
# http://wiki.nginx.org/QuickStart
# http://wiki.nginx.org/Configuration
#
# Generally, you will want to move this file somewhere, and start with a clean
# file but keep this around for reference. Or just disable in sites-enabled.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##
 
# Default server configuration
#
server {
        listen 80 default_server;
        listen [::]:80 default_server;
 
        server_name openfoodfacts.ovh *.openfoodfacts.ovh;
 
        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # Self signed certs generated by the ssl-cert package
        # Don't use them in a production server!
        #
        # include snippets/snakeoil.conf;
 
        root /home/off/html;
 
        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;
 
        location ~ ^/(images|js|rss|data|files|resources|foundation)/ {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }
 
        location = /robots.txt {
                try_files $uri $uri/ =404;
        }
 
        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://127.0.0.1:8001/cgi/display.pl?;
        }
 
        location /cgi/ {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://127.0.0.1:8001;
        }
 
        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #      deny all;
        #}
}
 
 
</pre>
 
= Old notes =


All instructions below are for user "stephane", please use your own name. :-)
All instructions below are for user "stephane", please use your own name. :-)
Line 33: Line 399:
Create 2 A records in the DNS zone of the domain you want to use:
Create 2 A records in the DNS zone of the domain you want to use:
* openfoodfacts.ovh 94.23.195.82
* openfoodfacts.ovh 94.23.195.82
** *.openfoodfacts.ovh 94.23.195.82
* *.openfoodfacts.ovh 94.23.195.82


== Install ==
== Install ==
Line 105: Line 471:


=== Apache 2.2 + mod_perl: Product Opener backend ===
=== Apache 2.2 + mod_perl: Product Opener backend ===
==== Build ====


<pre>
<pre>
Line 118: Line 486:
make
make
make install
make install
cd ~


cd ~/src
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.8.tar.gz  
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.8.tar.gz  
tar xvfz mod_perl-2.0.8.tar.gz
tar xvfz mod_perl-2.0.8.tar.gz
Line 125: Line 495:
make
make
make install
make install
cd ~
</pre>
==== Configuration ====
In ~/apache/conf/httpd.conf:
Change the port to something else than 80 (the reverse proxy will listen on 80 and forward to the backend on another port).
<pre>
Listen 8001
</pre>
If you are installing several Product Openers on the same machine, use different ports.
Change the Apache user to your user:
<pre>
User stephane
Group stephane
</pre>
</pre>
Add to apache/conf/httpd.conf:
<pre>
LoadModule perl_module modules/mod_perl.so
PerlWarn Off
PerlRequire /home/stephane/product-opener/cgi/startup.pl


<Location /cgi>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
Options +ExecCGI
Order allow,deny
Allow from all
</Location>
<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName openfoodfacts.ovh
ErrorLog /home/stephane/logs/error_log
CustomLog /home/stephane/logs/access_log combined
ScriptAlias /cgi/ "/home/stephane/product-opener/cgi/"
</VirtualHost>
PerlPostReadRequestHandler My::ProxyRemoteAddr
</pre>


=== Apache 2.4 - light weight reverse proxy ===
=== Apache 2.4 - light weight reverse proxy ===
==== Build ====


<pre>
<pre>
Line 143: Line 564:
tar xvfz apr-1.5.1.tar.gz
tar xvfz apr-1.5.1.tar.gz
tar xvfz apr-util-1.5.4.tar.gz  
tar xvfz apr-util-1.5.4.tar.gz  
ln -s apr-1.5.1 ap
ln -s apr-1.5.1 apr
ln -s apr-util-1.5.4 apr-util
ln -s apr-util-1.5.4 apr-util
cd ..  
cd ..  
Line 150: Line 571:


make
make
make install
sudo make install
cd ~
</pre>
</pre>


==== Configuration ====
in ~/proxy/conf/httpd.conf :
* uncomment #LoadModule rewrite_module modules/mod_rewrite.so
* uncomment #LoadModule slotmem_shm_module modules/mod_slotmem_shm.so


in proxy/conf/httpd.conf :
Add in httpd.conf :
    uncomment #LoadModule rewrite_module modules/mod_rewrite.so
    uncomment #LoadModule slotmem_shm_module modules/mod_slotmem_shm.so


Add:
<pre>
<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName world.openfoodfacts.ovh
ServerAlias *.openfoodfacts.ovh


ErrorLog /home/stephane/logs/proxy_error_log
CustomLog /home/stephane/logs/proxy_access_log combined
ServerAdmin stephane@openfoodfacts.org


<Location "/foundation/foundation-icons" >
Header set Access-Control-Allow-Origin "*"
</Location>


<Location "/images/products" >
Header set Link "<http://creativecommons.org/licenses/by-sa/3.0/>; rel='license'; title='CC-BY-SA 3.0'"
</Location>
<Directory "/home/stephane/product-opener/html">
    Options Indexes FollowSymLinks
    Require all granted
</Directory>
ProxyPreserveHost On
RewriteEngine on
RewriteCond  %{REQUEST_URI}  !/./
RewriteRule ^(/cgi/.*)$ http://localhost:8001$1 [P,L]
RewriteMap escape int:escape
RewriteRule ^/favicon.ico$ /favicon.ico [L]
RewriteCond  %{REQUEST_URI}  !^/images/
RewriteCond  %{REQUEST_URI}  !^/js/
RewriteCond  %{REQUEST_URI}  !^/rss/
RewriteCond  %{REQUEST_URI}  !^/robots
RewriteCond  %{REQUEST_URI}  !^/clicks/
RewriteCond  %{REQUEST_URI}  !^/data/
RewriteCond  %{REQUEST_URI}  !^/files/
RewriteCond  %{REQUEST_URI}  !^/resources/
RewriteCond  %{REQUEST_URI}  !^/foundation/
RewriteRule  ^(.*)$ http://localhost:8001/cgi/display.pl?${escape:$1} [P,L,QSA]
</VirtualHost>
<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName openfoodfacts.ovh
ErrorLog /home/stephane/logs/proxy_error_log
CustomLog /home/stephane/logs/proxy_access_log combined
DirectoryIndex index.html index.shtml
<Directory "/home/stephane/product-opener/html">
    Options -Indexes +FollowSymLinks +Includes
    Require all granted
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} ^stephane\.openfoodfacts\.ovh
RewriteRule ^/products$ /products.shtml [L]
</VirtualHost>
</pre>


=== Install Perl modules ===
=== Install Perl modules ===
Line 172: Line 652:
apt-get install libcache-memcached-fast-perl libjson-perl libclone-perl  libgraphviz-perl  
apt-get install libcache-memcached-fast-perl libjson-perl libclone-perl  libgraphviz-perl  
apt-get install libmime-lite-perl  # needs to be done after installing exim4
apt-get install libmime-lite-perl  # needs to be done after installing exim4
apt-get install libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl libmongodb-perl liburi-find-perl
apt-get install libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl
apt-get install libxml-feedpp-perl libmongodb-perl liburi-find-perl
</pre>
</pre>


Line 183: Line 664:
</pre>
</pre>


Modules that HAVE to be installed using cpan:
As of Feb 3rd 2015, the MongoDB perl module installed through "apt-get install libmongodb-perl" is too old (0.45):
<pre>
perl -MMongoDB -e 'print $MongoDB::VERSION . "\n"'
0.45
</pre>
This causes an error when trying to access a list of "tags" such as categories:
<pre>
Software error:
Not an ARRAY reference at /home/stephane/product-opener/cgi/Blogs/Display.pm line 797.
</pre>
A newer MongoDB perl module needs to be installed with cpan.
<pre>
cpan
install MongoDB
</pre>
<pre>
perl -MMongoDB -e 'print $MongoDB::VERSION . "\n"'
v0.708.0.0
</pre>
cgi/startup.pl preloads a lot of modules, you can check that those modules are installed by running:
<pre>
cd /home/stephane/product-opener/cgi
perl startup.pl
</pre>


=== Create some paths and links ===
=== Create some paths and links ===
Line 190: Line 705:
ln -s ./ Blogs
ln -s ./ Blogs
ln -s S
ln -s S
</pre>
=== Update IP to country GeoIP database ===
Product Opener uses the free legacy GeoIP Lite database from Maxmind:
* http://search.cpan.org/~maxmind/Geo-IP-1.45/lib/Geo/IP.pm
* http://dev.maxmind.com/geoip/legacy/geolite/
The database installed with apt-get install geoip-database can be very old.
To update manually, download the new databases to:
<pre>
/usr/share/GeoIP# ls -lrt
total 7256
-rw-r--r-- 1 root root 1103675  5 déc.  2013 GeoIPv6.dat
-rw-r--r-- 1 root root  600352  5 déc.  2013 GeoIP.dat
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
gzip -d GeoIP.dat.gz
</pre>
Retrieved from "https://test-wiki.openfoodfacts.org/Special:MobileDiff/1098...22042"