123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368 |
- README scopus
- Ak, 27.3.2017
- Get information from Scopus database.
- This queries work only with access to Scopus (e.g. from KIT LAN)
- Scopus service is not public vailable.
- Content
- info Documentation, website, etc
- readme.txt This file
- my_scopus.py List of scopus author ids
- ak_scopus.py Functions to access scopus
- ak_wordpress.py Functions to creates Wordpress posts + comments
- scopus-get-publications.py Script to query Scopus
- test-scopus.py Application with some functions to get publication entries
- Prints a list with some formatting
- test-scopus2.py Example from one of the website, only one query
- Usage:
- 1. Go to Scopus and retrieve the scopus author ids for the scientists in your group.
- Define the ids in my_scopus.py and group them.
- 2. Select one of more author groups in scopus-get-publications.py (main part at
- the end of the file). Check definition of database and wordpress installation.
- 3. Execute scopus-get-publications.py.
- python -W ignore scopus-get-publications.py
- Note: The -W ignore flag might be necessary if the INSERT IGNORE causes warnings.
- Example run:
- ufo:~/scopus # python -W ignore scopus-get-publications.py
- ***********************************************
- **** scopus-get-publications / 2017-03-27 *****
- ***********************************************
- === Update of publications for the author group: Computing
- Total number of publications: 54
- === Update of publications for the author group: X-ray Imaging
- Total number of publications: 39
- === Update of publications for the author group: Electronics
- Total number of publications: 132
- === Update of publications for the author group: Morphology
- Total number of publications: 21
- === Create posts for newly registered publication in scopus
- Nothing new found
- === Update citatation of all publication in the database
- Total number of publications is 281
- === Create comments for newly registered citations in scopus
- Number of new citations is 0
- Summary: (see also logfile /root/scopus/scopus-publications.log)
- Date = 2017-03-27 21:28:36.002624
- NPubs = 281
- NNewPubs = 0
- NCites = 4699
- NNewCites = 0
- Runtime = 0:00:11.496362
- Further enhancements
- Todo:
- - Reprocessing of all post, if the format has changed
- E.g. add button
- with Email to author or a new category has been added
- - Query only the latest citations for each publications not all.
- - Store JSON-Data of all publications
- - Get bibliographic information for display at the web page of a reseach group
- like UFO or may be also later for the DTS program.
- - Handle wrong publications in scopus for author with same name
- - Automatically include reports and student thesis by
- bibtex definition and upload on a server!?
- -> Would have the nice effect, that all student work is organized systematically!!!
- Structure of the database
- Both tables keep the reference to the publications in Scopus and the
- Wordpress ids. With this information, reprocessing is possible (but not
- implemented now).
- Table publocations:
- MariaDB [scopus]> describe publications;
- +--------------+--------------+------+-----+---------+----------------+
- | Field | Type | Null | Key | Default | Extra |
- +--------------+--------------+------+-----+---------+----------------+
- | id | int(11) | NO | PRI | NULL | auto_increment |
- | scopusid | varchar(255) | YES | UNI | NULL | |
- | wpid | int(11) | YES | | NULL | |
- | citedbycount | int(11) | YES | | NULL | |
- | citesloaded | int(11) | YES | | NULL | |
- | categories | varchar(255) | YES | | NULL | |
- | doi | varchar(255) | YES | | NULL | |
- | title | varchar(255) | YES | | NULL | |
- | abstract | text | YES | | NULL | |
- | bibtex | text | YES | | NULL | |
- | ts | datetime | YES | | NULL | |
- | scopusdata | text | YES | | NULL | |
- | eid | varchar(255) | YES | | NULL | |
- +--------------+--------------+------+-----+---------+----------------+
- Table citations:
- MariaDB [scopus]> describe citations;
- +--------------+--------------+------+-----+---------+----------------+
- | Field | Type | Null | Key | Default | Extra |
- +--------------+--------------+------+-----+---------+----------------+
- | id | int(11) | NO | PRI | NULL | auto_increment |
- | scopusid | varchar(255) | YES | | NULL | |
- | eid | varchar(255) | YES | | NULL | |
- | wpid | int(11) | YES | MUL | NULL | |
- | wpcommentid | int(11) | YES | | NULL | |
- | citedbycount | int(11) | YES | | NULL | |
- | citesloaded | int(11) | YES | | NULL | |
- | categories | varchar(255) | YES | | NULL | |
- | doi | varchar(255) | YES | | NULL | |
- | scopusdata | text | YES | | NULL | |
- | title | varchar(255) | YES | | NULL | |
- | abstract | text | YES | | NULL | |
- | bibtex | text | YES | | NULL | |
- | ts | datetime | YES | | NULL | |
- +--------------+--------------+------+-----+---------+----------------+
- Setup of scopus database in mysql
- create database scopus;
- CREATE USER 'scopus@localhost';
- grant all on scopus.* to 'scopus'@'localhost' identified by '$scopus$';
- # create tables
- mysql -u scopus -p scopus < create_scopus.sql
- Publications in Scopus:
- Sometime (unfortunately quite often) a author id in Scopus is not unique but
- identifies several researchers with the same name. E.g. Michele Caselle (3 persons)
- Matthias Balzer (2).
- This case is currently handled manually by deleting all publications from the unknown
- authors. Might be possible to implement also a black list??
- Sample data from Scopus:
- {
- "abstracts-retrieval-response": {
- "authors": {
- "author": [
- {
- "@_fa": "true",
- "@auid": "15076530600",
- "@seq": "1",
- "affiliation": {
- "@href": "http://api.elsevier.com/content/affiliation/affiliation_id/60102538",
- "@id": "60102538"
- },
- "author-url": "http://api.elsevier.com/content/author/author_id/15076530600",
- "ce:given-name": "Suren",
- "ce:indexed-name": "Chilingaryan S.",
- "ce:initials": "S.",
- "ce:surname": "Chilingaryan",
- "preferred-name": {
- "ce:given-name": "Suren",
- "ce:indexed-name": "Chilingaryan S.",
- "ce:initials": "S.",
- "ce:surname": "Chilingaryan"
- }
- },
- {
- "@_fa": "true",
- "@auid": "35313939900",
- "@seq": "2",
- "affiliation": {
- "@href": "http://api.elsevier.com/content/affiliation/affiliation_id/60102538",
- "@id": "60102538"
- },
- "author-url": "http://api.elsevier.com/content/author/author_id/35313939900",
- "ce:given-name": "Andreas",
- "ce:indexed-name": "Kopmann A.",
- "ce:initials": "A.",
- "ce:surname": "Kopmann",
- "preferred-name": {
- "ce:given-name": "Andreas",
- "ce:indexed-name": "Kopmann A.",
- "ce:initials": "A.",
- "ce:surname": "Kopmann"
- }
- },
- {
- "@_fa": "true",
- "@auid": "56001075000",
- "@seq": "3",
- "affiliation": {
- "@href": "http://api.elsevier.com/content/affiliation/affiliation_id/60032633",
- "@id": "60032633"
- },
- "author-url": "http://api.elsevier.com/content/author/author_id/56001075000",
- "ce:given-name": "Alessandro",
- "ce:indexed-name": "Mirone A.",
- "ce:initials": "A.",
- "ce:surname": "Mirone",
- "preferred-name": {
- "ce:given-name": "Alessandro",
- "ce:indexed-name": "Mirone A.",
- "ce:initials": "A.",
- "ce:surname": "Mirone"
- }
- },
- {
- "@_fa": "true",
- "@auid": "35277157300",
- "@seq": "4",
- "affiliation": {
- "@href": "http://api.elsevier.com/content/affiliation/affiliation_id/60102538",
- "@id": "60102538"
- },
- "author-url": "http://api.elsevier.com/content/author/author_id/35277157300",
- "ce:given-name": "Tomy",
- "ce:indexed-name": "Dos Santos Rolo T.",
- "ce:initials": "T.",
- "ce:surname": "Dos Santos Rolo",
- "preferred-name": {
- "ce:given-name": "Tomy",
- "ce:indexed-name": "Dos Santos Rolo T.",
- "ce:initials": "T.",
- "ce:surname": "Dos Santos Rolo"
- }
- },
- {
- "@_fa": "true",
- "@auid": "35303862100",
- "@seq": "5",
- "affiliation": {
- "@href": "http://api.elsevier.com/content/affiliation/affiliation_id/60102538",
- "@id": "60102538"
- },
- "author-url": "http://api.elsevier.com/content/author/author_id/35303862100",
- "ce:given-name": "Matthias",
- "ce:indexed-name": "Vogelgesang M.",
- "ce:initials": "M.",
- "ce:surname": "Vogelgesang",
- "preferred-name": {
- "ce:given-name": "Matthias",
- "ce:indexed-name": "Vogelgesang M.",
- "ce:initials": "M.",
- "ce:surname": "Vogelgesang"
- }
- }
- ]
- },
- "coredata": {
- "citedby-count": "0",
- "dc:description": "X-ray tomography has been proven to be a valuable tool for understanding internal, otherwise invisible, mechanisms in biology and other fields. Recent advances in digital detector technology enabled investigation of dynamic processes in 3D with a temporal resolution down to the milliseconds range. Unfortunately it requires computationally intensive recon- struction algorithms with long post-processing times. We have optimized the reconstruction software employed at the micro-tomography beamlines at KIT and ESRF. Using a 4 stage pipelined architecture and the computational power of modern graphic cards, we were able to reduce the processing time by a factor 75 with a single server. The time required to reconstruct a typical 3D image is reduced down to several seconds only and online visualization is possible for the first time.Copyright is held by the author/owner(s).",
- "dc:identifier": "SCOPUS_ID:84859045029",
- "dc:title": "Poster: A GPU-based architecture for real-time data assessment at synchrotron experiments",
- "link": [
- {
- "@_fa": "true",
- "@href": "http://api.elsevier.com/content/abstract/scopus_id/84859045029",
- "@rel": "self"
- }
- ],
- "prism:aggregationType": "Conference Proceeding",
- "prism:coverDate": "2011-12-01",
- "prism:doi": "10.1145/2148600.2148624",
- "prism:pageRange": "51-52",
- "prism:publicationName": "SC'11 - Proceedings of the 2011 High Performance Computing Networking, Storage and Analysis Companion, Co-located with SC'11",
- "prism:url": "http://api.elsevier.com/content/abstract/scopus_id/84859045029"
- }
- }
- }
- Installation of python, mysql et al:
- pip install python-wordpress-xmlrpc
- Konfiguration Webserver (muss man wohl nach jeder Installation neu machen!!!)
- /etc/apache2/httpd.conf:
- LoadModule userdir_module libexec/apache2/mod_userdir.so
- LoadModule php5_module libexec/apache2/libphp5.so
- Include /private/etc/apache2/extra/httpd-userdir.conf
- /etc/apache2/extra/httpd-userdir.conf:
- Include /private/etc/apache2/users/*.conf
- /etc/php.ini:
- pdo_mysql.default_socket= /tmp/mysql.sock
- mysql.default_socket = /tmp/mysql.sock
- mysqli.default_socket = /tmp/mysql.sock
- sh-3.2# apachectl restart
- Install website:
- Create archive with wp dublicator
- Save scopus database
- mysqldump -u scopus -p scopus > scopus-170322.sql
- Create database on remote system
- mysql:
- CREATE USER 'scopus'@'localhost' IDENTIFIED BY '$scopus$';
- GRANT ALL PRIVILEGES ON scopus.* TO 'scopus'@'localhost';
- CREATE DATABASE scopus;
- mysql -u scopus -p scopus < scopus-170322.sql
- Create database wp_ufo2;
- CREATE USER ‘ufo’@‘localhost' IDENTIFIED BY '$ipepdv$';
- GRANT ALL PRIVILEGES ON wp_ufo2.* TO ‘ufo’@‘localhost';
- CREATE DATABASE wp_ufo2;
- Import WP archive:
- mkdir ufo2
- chown -R wwwrun:www ufo2
- http://ufo.kit.edu/ufo2/installer.php
- Error: PHP module ZipArchive is missing
- Manual extraction is available in the advanced options !!!
- Installation Scopus-Scripts:
- pip install requests
- pip install python-wordpress-xmlrpc
- pip install pymysql
- Check configurations:
- scopus-get-piblications.py
- ak_wordpress.py
|