An attempt to map the areas with active conflict in Ukraine using open source twitter data.

Overview

Contributors Forks Stargazers Issues LinkedIn


Logo

Live Action Map (LAM)

An attempt to use open source data on Twitter to map areas with active conflict. Right now it is used for the Ukraine-Russia conflict, but in the future I hope it can be used for all sorts of dangerous situations.
Report Bug · Add Feature · Website Live! · Join Discord!

Table of Contents
  1. About The Project
  2. Getting Started
  3. Usage
  4. Roadmap
  5. Contributing
  6. License

About The Project

There are many twitter accounts posting live tweets about locations with conflicts. However, it is difficult to keep track of the locations especially with multiple different sources pointing out different location every few minutes. To make sure people can stay safe and take care of themselves, I have aggregated all the tweets into a single map that is easily accessible.

This project is a work in progress. I am working on adding more features and improving the map.

Website Link Image

How it works:

  • Tweets are sourced using keywords, hashtags and prepositions, such as the phrase "shooting... near ... location".
  • Tweets can also be sourced from known twitter accounts by passing their usernames.
  • Tweets are parsed with NLP and the location is extracted from the tweet, this however is not perfect so we need to filter locations later on.
  • Some tweets might talk about other countries reactions like "The US.." or "Russia.." or "Moscow..", in that case we remove all the locations that are not in Ukraine.
  • Some tweets might talk about multiple locations like "Shooting near the location and the location". In that case both locations are added to the map. Multiple markers can be added to the same location.
  • Finally we add markers for each tweet.
  • Markers will cluster together when you zoom out.
  • A single marker looks like a red pin on a map.
  • A cluster appears as a circle with a number inside it, the color shifts from green to orange to red depending on the number of markers in the cluster.
  • We are not taking data directly because that may be vulnerable to trolling and spamming.
  • We are using the Twitter v2 API to get the tweets, however it does not support parsing location directly from tweets.

(back to top)

Getting Started

To get a local copy up and running follow these simple example steps.

Prerequisites

  • Python
  • tweepy
  • spaCy
  • folium
  • geopy
  • tqdm
  • geography3 (optional, needed for experimental feature)

Installation

Python

  1. Get a free twitter Bearer Token from developer.twitter.com. Remember to create a new app and get the bearer token.
  2. Clone the repo
    git clone https://github.com/kinshukdua/LiveActionMap.git
  3. Install all prerequisites
    pip install -r requirements.txt
  4. Download en_core_web, for more info see --> explosion/spaCy#4577
     python3 -m spacy download en_core_web_sm
  5. Create a .env file based on the .env.example
    cp .env.example .env
  6. Set the Twitter bearer token to your own in the .env file created in the previous step.

Docker

  1. Get a Twitter Bearer Token
  2. Download the docker-compose.yaml-file
    wget https://raw.githubusercontent.com/kinshukdua/LiveActionMap/main/docker/docker-compose.yaml
  3. Create a .env file based on the .env.example
    wget https://raw.githubusercontent.com/kinshukdua/LiveActionMap/main/.env.example -O .env 
  4. Start the stack
    docker-compose up -d
    

(back to top)

Usage

Simply edit hashtags, prepositions and keywords and run scrape.py.

python scrape.py

(back to top)

Roadmap

  • Add tweet scraping
  • Add map
  • Add map clustering
  • Create a server to host the generated map
  • Add better filtering
  • Add tweet link on map
  • Use NLP to indicate danger level
  • Add misinformation prevention algorithm
  • Multi-language Support
    • Ukranian
    • Russian

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Comments
  • Add CI/CD and Tests

    Add CI/CD and Tests

    As the project grows, we need a way to make sure it is easy to test incoming PRs instead of me manually pulling and testing everything. The best way would be to setup a CI/CD pipeline using Github Actions. At the present moment we don't have any tests except the one introduced in #21. Ideally I would like to setup tests and then a CI/CD pipeline. However, the server running the website right now, is manually using cron jobs. I would instead like to wait for #16 to finish and then create a pipeline with the docker image.

    Meanwhile, feel free to add tests and help me write a CI pipeline.

    enhancement 
    opened by kinshukdua 14
  • UI Changes and wider search

    UI Changes and wider search

    Ok, so I wish to contribute to this project. I made some minor changes, here they are:

    1. Cap tweet preview for the folium marker preview to 100 letters.
    2. Add one more hashtag for tweepy to search (#OSINT).
    3. Adjust default coordinates and zoom.
    4. Change tileset to something more fitting (in my opinion).
    5. Change User Agent which geopy uses for Nominatim.

    Please inform me of anything wrong I did (this is my first pr).

    opened by sahal-mulki 9
  • fix ci, use curl to trigger gitlab deploy

    fix ci, use curl to trigger gitlab deploy

    instead of using a third-party GitHub action.

    #52 I've used just the token secret, the gitlab ref name is hardcoded to master.

    I haven't tested it, as I'd need the secret value and gitlab access.

    opened by edubacco 4
  • Implement a ui

    Implement a ui

    I think it would be a pretty good idea to implement a ui for this instead of just generating a PNG.

    If needed i could spare some time in Vue. By all means i have a free weekend right now.

    opened by DomiiBunn 4
  • Add blacklist twitter userIDs

    Add blacklist twitter userIDs

    Fix #5

    Excluding accounts that spreads misinformation

    This pullrequest added:

    • [x] Changes in README (adding point 4)

    image

    • [x] Adds new env variable to handle blacklisted userIDs
    • [x] Add test folder and some test coverage of the implemented feature
    opened by GabrieleCalarota 3
  • Catch nomatim api error/timeout

    Catch nomatim api error/timeout

    Don't exit the whole application if fetching geolocation data fails. It will look like this:

    Adding markers... (This may take a while)
     10%|█████████████████▌                                       | 50/512 [01:12<14:03,  1.83s/it] 
    Getting geolocation data for the South China Sea failed
     11%|███████████████████                                      | 55/512 [01:26<19:59,  2.63s/it] 
    
    opened by laurin 2
  • Domain name

    Domain name

    I was trying to get a domain on freenom, but it doesn't let me register, for the time being I'll deploy it on my custom domain but if anyone has a spare domain lying around or if they can register a free "liveactionmap" domain on freenom that would be great.

    opened by kinshukdua 2
  • Remove temporary tweets file after every x minutes.

    Remove temporary tweets file after every x minutes.

    Since we're no longer dependent on the cronjob, we need to remove the tweets in the python code itself. However deleting the file after each run as done in an earlier PR is not useful because there's too few data points, we can try to increase the number of results we get from Twitter but that's a separate issue.

    Also let's make sure the schedule time for the scraping and removing tweets don't overlap (they are not multiples) otherwise the tweets file might be deleted while the bot is running outputting an empty map.

    opened by kinshukdua 1
  • Backend API

    Backend API

    Moving to a dynamic backend should be it's own issue

    Originally posted by @Krishna-Sivakumar in https://github.com/kinshukdua/LiveActionMap/issues/24#issuecomment-1053660218

    enhancement 
    opened by DomiiBunn 1
  • Self-host a Nominatim Service

    Self-host a Nominatim Service

    Self-hosting a Nominatim service will help us cut down on time wasted doing API calls. Map building takes the most time. And right now, most of the time while building the map is spent on doing 2 API calls to a public Nominatim service per tweet, which adds a lot of latency.

    Since we already have a docker build, it doesn't hurt to add one or two more containers to our deployment.

    Links to refer: https://github.com/mediagis/nominatim-docker https://nominatim.org/release-docs/latest/admin/Installation/

    This will help us shift to a dynamic backend in the long run.

    enhancement 
    opened by Krishna-Sivakumar 1
  • API: Zones

    API: Zones

    Create a GET API serving /api/zonesproviding a response containing all currently available markers.

    Expected response

    [
        {
            "positions": [
                [
                    50.36429316995319,
                    30.228621662109
                ],
                [
                    50.51303377189189,
                    30.28741424805162
                ],
                [
                    50.60122461757218,
                    30.787151228563896
                ],
                [
                    50.44548234821721,
                    30.77081995469095
                ]
            ],
            "color": "green",
            "title": "Safe Zone",
            "content": "Some safe zone over here"
        }
    ]
    
    enhancement 
    opened by DomiiBunn 0
  • Add a .onion link

    Add a .onion link

    This is so people can access the site using tor, might make some people feel safer.

    This would need a separate frontend that doesn't contain any tracking and no links to anything but out service

    enhancement 
    opened by DomiiBunn 5
  • API: Markers

    API: Markers

    Create a GET API serving /api/markers providing a response containing all currently available markers.

    Expected response

    {
        "markers": [
            {
                "position": [
                    49.038230248475905,
                    31.450182690663947
                ],
                "title": "Twit Title with image",
                "content": "",
                "user": "Twitter User",
                "uri": "https://twitter/postUrl",
                "image": "imageURL",
                "timestamp": "unixtimestamp of tweet"
            },
            {
                "position": [
                    48.038230248475905,
                    31.450182690663947
                ],
                "title": "Twit Title without image",
                "content": "Twit Text",
                "user": "Twitter User",
                "uri": "https://twitter/postUrl",
                "timestamp": "unixtimestamp of tweet"
            }
        ],
        "timestamp": "unixTimeStamp"
    }
    
    enhancement 
    opened by DomiiBunn 2
  • Analitics

    Analitics

    It would be neat to see how many hits the site gets.

    I'd recommend either google or unami(this is more gdpr complaint and can be self hosted on Heroku for free)

    enhancement 
    opened by DomiiBunn 2
  • Dynamic Fronted

    Dynamic Fronted

    How about I make a quick mockup with vuejs that would just read an api. That way once the project is evolved enough it would be enough to just hook one and the other together.

    Originally posted by @Krishna-Sivakumar in https://github.com/kinshukdua/LiveActionMap/issues/24#issuecomment-1053660218

    opened by DomiiBunn 3
A project that analyses COVID-19 and restaurant-related data across Toronto Forward Sortation Areas and compares them across a variety of metrics.

restaurant-covid-analysis This project was created to analyze the relationship between restaurants in Toronto and COVID-19 cases across forward sortat

Benjamin Panet 1 Oct 20, 2022
Using the Traveling Salesman Problem algorithm to calculate the shortest route possible through Ukraine's 50 largest cities.

Using the Traveling Salesman Problem algorithm to calculate the shortest route possible through Ukraine's 50 largest cities. Used Google's TSP resources and distance API's. Applications of project can be used to optimize routes travelled to rescue refugees or deliver supplies.

Stuti Arora 1 Jun 13, 2022
Dataset of air raid sirens in the Ukraine-Russia war

Air Raid Dataset Russia invaded Ukraine on February 24, 2022. This repository contains dataset file air-raid-dataset.csv with information about the ai

Vadym Klymenko 33 Nov 25, 2022
Minecraft:Hacked is a video series exploring various technical areas of Minecraft.

Minecraft:HACKED Minecraft:HACKED is a video series by @LiveOverflow exploring various technical areas of Minecraft.It's a weird series trying to comb

null 183 Nov 22, 2022
The currency sale rate information getter. Work with National Bank of Ukraine and PrivatBank.

nbu-privat-currency-sale: currency sale information getter from National Bank of Ukraine and PrivatBank What that library can do? nbu-privat-currency-

Назар 0 May 17, 2022
Count the number of people downstream of areas of interest.

downstream-beneficiaries-cli Count the number of people downstream of pixels of interest. Installation conda create -p ./env python=3.9 -c conda-forge

James Douglass 1 May 9, 2022
Air-raid sirens in Ukraine monitor device

Air Raid Monitor for Raspberry Pi This device shows the current status of air raid sirens in Ukraine. The data is pulled every 10 seconds from https:/

Dmytro Panin 60 Oct 22, 2022
Tool to help those in Ukraine find alternate escape routes. Made for my friend Vic who is currently there.

FlixBus-Scraper Tool to help those in Ukraine find alternate escape routes. Made for my friend Vic who is currently there. Usage Use the script as fol

Jordan Jay 4 Aug 25, 2022
A tool to inform the general Russian population about what's going on in year 2022 on Ukraine via posting 5-star reviews to Yandex.

yandex-conqueror A tool to inform the general Russian population about what's going on in year 2022 on Ukraine via posting 5-star reviews to Yandex. L

Piotr Maślanka 6 Mar 16, 2022
A tool to magnify the local areas of images.

Image_Local_Magnification_Tool A tool to magnify the local areas of images. Requirements: PyQt5, numpy, cv2 Usage: run main.bat on Windows or python m

Juewen Peng 8 Oct 29, 2022
A simple tool to change areas of certain colours in images.

Image Change-Colour Tool This is a simple tool to change the colour of an image. It is written in Python 3 and uses the Pillow library. Usage Install

null 1 Oct 1, 2022
chemical analyse tools for converting SMILES to map structure, or converting the map structure to SMILES

SMILES Mapper convert SMILES to map structure, or convert the map structure to SMILES Usage # in the parent folder of src python ./src/main.py First

irumeria 1 Jun 15, 2022
Built for the LMN3, an open source DAW-in-a-box, this creates a simple GUI to map drum kit sounds to keys

LMN3-Drum-Kit-Mapper Built for the LMN3, an open source DAW-in-a-box, this creates a simple GUI to map drum kit sounds to keys Prerequisites This is p

null 4 Aug 12, 2022
This repository is an attempt at implementing the paper Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network using PyTorch.

ESPCN This repository is implementation of the "Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Ne

Gaurav 0 Oct 7, 2022
An attempt to keep a current copy of Genshin Impact's cooking recipe data

genshin-recipe-data A repository to keep a current copy of Genshin Impact's cooking recipe data. Updated on a best effort basis. Why? I'm making tools

null 1 Mar 30, 2022
Easy-to-use tool for mining tweets with Twitter API. Support both twitter API version 1 and academic API

Twitter_Crawler Easy-to-use tool for mining tweets with Twitter API. Support both Twitter API version 1 and academic API. It will collect tweets based

Hongbo Fang 12 Aug 22, 2022
Twitter Garbage Collection - python scripts which can be used to perform a cleanup of your Twitter account

This repository contains scripts which can be used to perform a cleanup of your Twitter account. It deletes all the tweets

Gopal Goel 1 Apr 22, 2022
This project was made for the FastAPI Course at Platzi. It's an API for twitter, but in this twitter you can edit a tweet.

Twiter API | FastAPI ⚡ This project was made for the FastAPI Course at Platzi. It's an API for twitter, but in this twitter you can edit a tweet. Run

Salmerón Alvarado 2 Oct 31, 2022
Simple Python script to fetches weather data from Open Weather Map by Adithya Krishnan.

Weather Fetcher Simple Python script to fetches weather data from Open Weather Map by Adithya Krishnan. Requirement API: http://openweathermap.org/app

Adithya Krishnan 1 Mar 22, 2022