Dear all,
I hope this message finds everybody well. First let me apologise for putting my message in the general category but I was unsure about where it fit.
I have the following issue: I am currently analysing the tweets from the timeline of the Greek Prime Minister regarding the Coronavirus epidemic. The main problem stems from the fact that the console will not display the most common terms in the tweets in a legible format. Instead it is some form of code, UTF-8 as I speculate.
First, I checked if the data had not been compromised by creating a xlsx file. I checked and everything is in order, and I was also successful in removing the stopwords via the stopword package.
Yet still, the problem remains. I am unable to display the characters in the console in Greek. My first attempt was to Reopen the file with encoding, and the UTF-8 format. Unfortunately it did not resolve anything. Therefore, I would greatly appreciate any advice on how to resolve this issue so that the Greek characters can be displayed, and I can visualise the most important terms. I have also found this link: https://tutorials.quanteda.io/import-data/encoding/ but the main problem is that twitter data are stored in JSON format.
If you request a reprex I can provide it along with any information that might help you. I apologise if I have not included it but I am not so experienced in coding so I do not know which additional information might be helpful.
I thank you in advance for your help and time.
Best regards,
MiltR
Hi, @MiltR,
Unicode can be tricky. It depends on how the source data was encoded and handled in the workflow.
It may be relatively simple to do the conversion, but for that some representative data is needed, and the current code that is generating the display. (Again, just enough to reproduce the problem.)
It's likely to be an encoding problem, since
print("γνωρίζω (gnorizo) / ξέρω (ksero) = to know")
#> [1] "γνωρίζω (gnorizo) / ξέρω (ksero) = to know"
Created on 2020-03-31 by the reprex package (v0.3.0)
I would like to apologise for the late reply mister Technocrat, but unfortunately I have been trying to create a reprex with no success at all. I get an error message despite my best efforts. I know that I can bypass that problem but you will be unable to reproduce my example. I am at a loss of what to do. Again I apologize for taking so long to reply, and thank you for your willingness to help me.
library(tidytext)
#> Warning: package 'tidytext' was built under R version 3.5.3
library(widyr)
#> Warning: package 'widyr' was built under R version 3.5.3
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library(quanteda)
#> Warning: package 'quanteda' was built under R version 3.5.3
#> Package version: 1.5.1
#> Parallel computing: 2 of 8 threads used.
#> See https://quanteda.io for tutorials and examples.
#>
#> Attaching package: 'quanteda'
#> The following object is masked from 'package:utils':
#>
#> View
library(dtplyr)
#> Warning: package 'dtplyr' was built under R version 3.5.3
library(tokenizers)
#> Warning: package 'tokenizers' was built under R version 3.5.3
prime_minister_tweets_clean %>%
select(text_full) %>%
head() %>%
dput(., control = NULL)
#> Error in eval(lhs, parent, parent): object 'prime_minister_tweets_clean' not found
prime_minister_tweets_clean %>%
select(text_full) %>%
head() %>%
View()
#> Error in eval(lhs, parent, parent): object 'prime_minister_tweets_clean' not found
Created on 2020-04-02 by the reprex package (v0.2.1)
No worries, @MiltR.
The two warnings are not a problem, right now. It likely means that that the installed version of R is lagging and it may be time to update it.
The error just means that prime_minister_tweets_clean isn't loaded yet. You can check by simply using the ls() command. If it doesn't show up, the function can't find it either. I found two recent tweets from the Greek Prime Minister's account
library(quanteda)
#> Package version: 2.0.1
#> Parallel computing: 2 of 4 threads used.
#> See https://quanteda.io for tutorials and examples.
#>
#> Attaching package: 'quanteda'
#> The following object is masked from 'package:utils':
#>
#> View
suppressPackageStartupMessages(library(dplyr))
library(dtplyr)
library(tokenizers)
prime_minister_tweets_clean <- structure(list(text_full = structure(1:2,
.Names =
c("tweet1", "tweet2"),
.Label =
c("Θέλω εκ μέρους του ελληνικού λαού να ευχαριστήσω τόσο το Ίδρυμα Ωνάση όσο και όλα τα ιδρύματα και όλους τους πολίτες που στηρίζουν την ελληνική πολιτεία, και βοηθούν ώστε να θωρακίσουμε αυτούς που βρίσκονται στην πρώτη γραμμή αντιμετώπισης του κορονοϊού.", "Οδηγίες για τους συμπολίτες μας που νοσούν στο σπίτι με ήπια συμπτώματα αλλά και για όσους τους φροντίζουν. Παρακολουθούμε την υγεία μας, παραμένουμε ενημερωμένοι, μένουμε ασφαλείς. Για περισσότερες πληροφορίες"),
class = "factor")),
class = "data.frame",
row.names =
c("tweet1", "tweet2"))
prime_minister_tweets_clean %>%
select(text_full) %>%
head() %>%
dput(., control = NULL)
#> list(1:2)
prime_minister_tweets_clean %>%
select(text_full)
#> text_full
#> tweet1 Θέλω εκ μέρους του ελληνικού λαού να ευχαριστήσω τόσο το Ίδρυμα Ωνάση όσο και όλα τα ιδρύματα και όλους τους πολίτες που στηρίζουν την ελληνική πολιτεία, και βοηθούν ώστε να θωρακίσουμε αυτούς που βρίσκονται στην πρώτη γραμμή αντιμετώπισης του κορονοϊού.
#> tweet2 Οδηγίες για τους συμπολίτες μας που νοσούν στο σπίτι με ήπια συμπτώματα αλλά και για όσους τους φροντίζουν. Παρακολουθούμε την υγεία μας, παραμένουμε ενημερωμένοι, μένουμε ασφαλείς. Για περισσότερες πληροφορίες
Created on 2020-04-01 by the reprex package (v0.3.0)
Do you get the same result?
Good afternoon,
I run your example and this is what it produced:
library(quanteda)
#> Warning: package 'quanteda' was built under R version 3.5.3
#> Package version: 1.5.1
#> Parallel computing: 2 of 8 threads used.
#> See https://quanteda.io for tutorials and examples.
#>
#> Attaching package: 'quanteda'
#> The following object is masked from 'package:utils':
#>
#> View
suppressPackageStartupMessages(library(dplyr))
library(dtplyr)
#> Warning: package 'dtplyr' was built under R version 3.5.3
library(tokenizers)
#> Warning: package 'tokenizers' was built under R version 3.5.3
prime_minister_tweets_clean2 <- structure(list(text_full = structure(1:2,
.Names =
c("tweet1", "tweet2"),
.Label =
c("???? ?? ?????? ??? ????????? ???? ?? ??????????? ???? ?? ?????? ????? ??? ??? ??? ?? ???????? ??? ????? ???? ??????? ??? ????????? ??? ???????? ????????, ??? ??????? ???? ?? ??????????? ?????? ??? ?????????? ???? ????? ?????? ????????????? ??? ?????????.", "??????? ??? ???? ?????????? ??? ??? ?????? ??? ????? ?? ???? ?????????? ???? ??? ??? ????? ???? ??????????. ?????????????? ??? ????? ???, ??????????? ????????????, ??????? ????????. ??? ???????????? ???????????"),
class = "factor")),
class = "data.frame",
row.names =
c("tweet1", "tweet2"))
prime_minister_tweets_clean2 %>%
select(text_full) %>%
head() %>%
dput(., control = NULL)
#> list(1:2)
prime_minister_tweets_clean2 %>%
select(text_full)
#> text_full
#> tweet1 ???? ?? ?????? ??? ????????? ???? ?? ??????????? ???? ?? ?????? ????? ??? ??? ??? ?? ???????? ??? ????? ???? ??????? ??? ????????? ??? ???????? ????????, ??? ??????? ???? ?? ??????????? ?????? ??? ?????????? ???? ????? ?????? ????????????? ??? ?????????.
#> tweet2 ??????? ??? ???? ?????????? ??? ??? ?????? ??? ????? ?? ???? ?????????? ???? ??? ??? ????? ???? ??????????. ?????????????? ??? ????? ???, ??????????? ????????????, ??????? ????????. ??? ???????????? ???????????
Created on 2020-04-02 by the reprex package (v0.2.1)
Again, many thanks for your help.
Best regards,
M
Hello again,
I apologise for posting without a reply first. I thought that maybe this information is important. The Rstudio version I am using is 1.1.463. Also, I can provide any other information you may request. As I previously mentioned I stored the twitter data in an xlsx file, and I checked it. Everything is displayed properly, and I concluded that the data is not compromised in any way. Therefore, the culprit must be Rstudio.
Again, many thanks for your patience and your help.
Best regards,
M
Since it works on RStudio v.1.3, then the problem may be that the version that you are using is outdated. Try updating to the latest production version? (which will be 1.2.something)
Good evening,
I updated the R studio version and it now displays the characters in Greek after I view my data. I would like to apologise for troubling you for such a triviality, and thank you for taking the time to help me with that problem. Unfortunately my inexperience prevented me from looking at the simplest solution.
Best regards,
M
@MiltR, please don't feel bad. It's a process for everyone. You learn something by asking, and I learn something by looking for an answer, and now the solution will be available to the community in the search icon and someone else won't have to retrace our steps.
This has been a very friendly and supportive community for me, and please come back, even if you fear that the solution will be obvious in hindsight.
@technocrat, I actually encountered another problem related to stopwords. I had downloaded the package stopwords, and it successfully removed some of them, but others remained. The problem arose from the fact that there are tonal points in the Greek language, and a multitude of the stopwords did not include them.
Initially, I tried to include the stopwords in my own list, but R Studio would not recognise them. The problem was resolved by doing the following thing at the start of the session: Sys.setlocale("LC_CTYPE", "greek"). After that I re-run the chunk with my stopwords, and the number of words collected went from 71243 to 58026. I just thought that I should post this minor discovery here.
Most importantly, thank you both @technocrat and @dromano for taking the time to look at my problem and offer your assistance. And especially you @technocrat for your kind words. I truly appreciate the support from the community in its entirety, and this is the reason I will come back.
Best regards,
@MiltR
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.