Social Network Analysis using Large Language Models

42913 Social and Information Network Analysis @ FEIT UTS

Group Members

Project Report Keywords

Contents

Abstract. 3

1. Introduction. 3

2. Related Work and Motivation. 3

3. Dataset and Problem Formulation. 4

4. Label Construction. 4

5. Methodology: Feature-Based Machine Learning Pipeline. 5

5.1 Text and Time Features. 5

5.2 Graph Features from User Projection. 5

5.3 Baseline Models. 5

5.4 LLM-Based Behavioural Feature Augmentation. 6

6. Methodology: Bipartite Graph Neural Network. 6

6.1 Node and Edge Representations. 7

6.2 Message Passing Logic. 7

7. Experimental Design and Reproducibility. 7

8. Evaluation Metrics. 7

9. Results Reporting. 8

10. Discussion. 8

11. Novelty and Significance. 9

12. Limitations and Ethical Considerations. 9

13. Future Work. 10

14. Conclusion. 10

14. How to Reproduce the Project. 11

References. 11

Appendix A. Implementation Map. 11

Abstract

This report presents a fraudulent user detection framework for the Amazon Review dataset, using the Musical Instruments subset from the SNAP Amazon product graph. The task is formulated as a binary user classification problem: reviewers with more than 80% helpful votes are treated as benign users, while reviewers with fewer than 20% helpful votes are treated as fraudulent or suspicious users. The project combines text, temporal behaviour and network structure. First, a feature-based machine learning pipeline constructs reviewer-level variables from review text, posting timing and projected user-user graph statistics. Second, a graph neural network (GNN) models the original user-product bipartite graph directly, allowing user and product representations to exchange information through review edges. The implementation is designed for reproducibility through fixed configuration parameters, explicit preprocessing, deterministic train-test splitting and saved output files. The main contribution is not simply the use of a classifier, but the integration of text-attributed graph learning with fraud detection in an e-commerce context.

1. Introduction

Online reviews influence consumer decisions, seller reputation and product visibility. This creates an incentive for manipulation: coordinated reviewers may post misleading reviews, inflate ratings, or damage competitors. Traditional fraud detection methods often rely on tabular behaviour features such as review frequency, rating distribution and account age. However, review platforms are naturally networked systems. Users are connected to products through reviews, and suspicious activity can appear as abnormal connectivity, repeated co-reviewing patterns, or bursts of reviews on the same products.

The selected topic, Social Network Analysis using Large Language Models, is well matched to this problem because Amazon reviews contain both graph structure and rich textual content. Each user has written natural-language reviews, while each product is linked to many users. This makes the data a text-attributed graph: entities are connected through edges, and some entities or edges carry text. Recent research on text-attributed graph representation learning argues that language-model-derived textual representations can improve graph learning by capturing semantic information that simple handcrafted features miss. This report adapts that idea to fraudulent review detection.

The objective is to detect suspicious users using a binary classification setting. The project implements two complementary strategies. The first is an interpretable baseline pipeline using graph features, time features, text-style features and TF-IDF or sentence-transformer embeddings. The second is a bipartite GNN that preserves the user-product graph and learns user representations through message passing. Together, these methods provide a technically sound comparison between conventional machine learning and graph-based representation learning.

2. Related Work and Motivation

The project is motivated by two research directions. The first is text-attributed graph learning. He et al. (2024) study how explanations and language-model representations can enhance graph learning on networks where nodes have associated text. The key insight for this project is that text should not be treated as an afterthought: review content may contain signals about exaggeration, repetition, generic language or sentiment patterns. The second direction is group-based fraud detection. Yu et al. (2023) model e-commerce fraud as a graph problem and show that suspicious behaviour is often not isolated at the individual level; it may emerge from groups of entities connected through products, transactions or coordinated actions.

This project therefore combines individual-level behavioural modelling with network-level analysis. If a user has a low helpfulness ratio, posts many reviews in short bursts, shares products with many other low-helpfulness users, and uses repetitive or unusually stylised text, the combination of evidence is stronger than any individual signal. The implemented framework reflects this principle by creating multiple feature families and by training a GNN that exchanges information between user nodes and product nodes.

3. Dataset and Problem Formulation

The dataset used is the Amazon Musical Instruments review file from SNAP. The raw file is compressed as JSON lines, where each line corresponds to one product review. The main columns used by the project are reviewerID, asin, reviewText, helpful, overall and unixReviewTime. The reviewerID identifies the user, asin identifies the product, reviewText contains the natural-language review, helpful contains the pair of helpful votes and total votes, overall contains the rating, and unixReviewTime provides the review timestamp.

The graph is defined as a bipartite network G = (U, P, E), where U is the set of users, P is the set of products and E is the set of reviews. An edge exists when user u reviews product p. The edge can also carry attributes such as rating and review time. The classification target is defined only for user nodes. Product nodes are not classified, but they help propagate information because suspicious users may concentrate around specific products or review patterns.

4. Label Construction

The labelling strategy follows the assignment specification. For each reviewer, the code sums all helpful votes and total votes across that user’s reviews. It then computes a helpfulness ratio:

helpfulness_ratio = helpful_votes_sum / total_votes_sum

Users with a helpfulness ratio greater than 0.8 are labelled as benign (class 0). Users with a helpfulness ratio below 0.2 are labelled as fraudulent or suspicious (class 1). Users in the middle range are ambiguous because their behaviour is not clearly positive or negative under this proxy. In the classical ML pipeline, these grey users are removed from supervised training. In the GNN pipeline, grey users can be kept in the graph as unlabeled nodes so that they still contribute structural context, while only clearly labelled users are used for the training loss.

This labelling rule is practical because it allows supervised learning without manually annotated fraud labels. However, it is also a limitation. Low helpfulness is not the same as proven fraud; a review may be unhelpful because it is short, unpopular or written for a niche product. Therefore, the correct interpretation of the output is suspiciousness under the helpfulness proxy, not confirmed malicious intent.

5. Methodology: Feature-Based Machine Learning Pipeline

This work proposes a hybrid fraud detection framework based on feature engineering and graph-based learning. The pipeline integrates textual, temporal, and structural signals extracted from Amazon review data to detect suspicious reviewer behaviour. A bipartite Graph Neural Network (GNN) is used as the final predictive model, operating on enriched user and product representations.

The overall system consists of three main components:

1. Feature engineering (text, temporal, and behavioral signals) => LLM explanation
2. Graph construction and structural feature extraction
3. Downstream machine learning and GNN-based classification

Figure 1. Feature-based fraud detection pipeline implemented in main.py.

5.1 Text and Time Features

For each user, all reviews are aggregated into a single reviewer profile. Text-style features include average review length, token count, exclamation count, question count, uppercase ratio, repeated word ratio and a simple lexicon-based sentiment score. These variables are not intended to fully understand language, but they capture suspicious stylistic patterns such as excessive repetition, unusually short reviews, exaggerated punctuation or abnormal sentiment.

Temporal features measure how reviews are distributed over time. The project computes review count, active days, reviews per active day, maximum reviews in one day, burst ratio, mean gap between reviews, minimum gap, standard deviation of gaps and short-gap ratio. These features are important because coordinated or automated reviewing may produce bursts of activity, many reviews in a single day or unusually short time gaps between reviews.

5.2 Graph Features from User Projection

The project also constructs a projected user-user graph. Two users are connected if they reviewed the same product. The edge weight increases when users share more products, and an additional same-day shared variable captures whether they reviewed the same product on the same day. This projection is a compact way of representing reviewer coordination without training a full GNN.

From the projected graph, the implementation extracts user degree, weighted degree, same-day weight, clustering coefficient, PageRank, community identifier and community size. High degree or high same-day shared activity can indicate that a reviewer is part of a dense co-reviewing group. Community features are especially relevant to the group-based fraud detection perspective, where suspicious behaviour may emerge from clusters rather than isolated accounts.

5.3 Baseline Models

The feature-based pipeline trains several models to compare different sources of information. Logistic regression is used for graph/time features and text-style features because it is simple and interpretable. A random forest combines structured features and optional sentence-transformer embeddings to capture non-linear interactions. If transformer embeddings are not enabled, the code trains a TF-IDF plus graph logistic regression model, which provides a lightweight text baseline.

5.4 LLM-Based Behavioural Feature Augmentation

In addition to traditional feature engineering, this work integrates a Large Language Model (LLM) as an auxiliary reasoning module to enrich user-level representations.

Each user’s complete review history is provided to a pretrained LLM (LLaMA 3.2–3B), which analyses the reviewer’s behaviour holistically. The model is prompted to identify patterns indicative of suspicious activity based on linguistic and behavioural cues across multiple reviews.

The LLM produces two outputs:

• LLM_prediction: a structured label indicating whether the reviewer is likely benign or suspicious.
• LLM_explanation: a natural language justification describing observed anomalies such as repetitive language, excessive positivity, inconsistent sentiment, or unusual reviewing patterns.

These outputs are not used as standalone classifiers. Instead, they are incorporated as auxiliary features and concatenated with graph-based, temporal, and textual representations before being passed into the bipartite GNN.

This design allows the model to benefit from:

• semantic representations (sentence-transformer embeddings),
• structured behavioural signals (engineered features),
• relational graph structure (user–product interactions),
• and high-level interpretability signals provided by the LLM.

The approach aligns with recent research in LLM-assisted representation learning, where large language models are used as feature generators or weak reasoning modules to enhance downstream machine learning systems rather than replace them.

6. Methodology: Bipartite Graph Neural Network

The second approach keeps the original graph structure instead of flattening it into a table. The graph contains user nodes, product nodes, and review edges. User nodes are described by behavioural and textual features. Product nodes are described by product-level statistics such as average rating, rating standard deviation, review count, number of unique reviewers, and reviews per month. Edges contain review rating and normalised review time. The framework follows a semi-supervised node classification setting in which only confidently labelled users contribute to supervised loss computation, while ambiguous grey-zone users remain in the graph as unlabeled nodes participating in message passing. This allows the GNN to preserve structural context without forcing uncertain users into potentially noisy labels.

Figure 2. Bipartite GNN structure for user-product review data.

6.1 Node and Edge Representations

The GNN data builder creates user feature tensors, product feature tensors, edge-index tensors and edge-attribute tensors. User features include numeric behaviour features and sentence-transformer text embeddings. Product features summarise the review behaviour around each product. Each review edge carries two attributes: the normalised rating and the normalised timestamp. This design allows the model to learn not only who reviewed what, but also whether the interaction was unusually timed or unusually rated.

6.2 Message Passing Logic

The GNN first encodes users, products and edges into hidden vectors. It then performs two message-passing steps. In the user-to-product step, information from users and review edges is aggregated into product representations. In the product-to-user step, the updated product representations are sent back to users. Finally, a classifier predicts each user’s class. This approach is suitable for fraud detection because a suspicious user may be easier to identify after considering the products they review and the other users connected to those products.

The implemented GNN can be viewed as a custom bipartite message-passing network rather than a standard off-the-shelf architecture. This is a reasonable design choice for the assignment because the Amazon review network naturally has two entity types. Treating products and users separately avoids forcing the graph into a homogeneous structure too early.

7. Experimental Design and Reproducibility

The implementation is designed to be reproducible. Configuration parameters are centralised, including input path, output directory, sample size, random seed, graph sampling limits, sentence-transformer model, TF-IDF size and test split. The default sample size is 10,000 reviews, which makes experimentation faster while preserving the option to run the full dataset by setting the sample value to None.

The feature-based pipeline uses stratified train-test splitting to preserve the class distribution in both partitions. The GNN pipeline creates labelled indices, shuffles them using a fixed random generator and applies an 80/20 train-test split. Class weights are computed from the training labels to reduce the effect of class imbalance. This is important because fraudulent and benign users may not appear in equal numbers after the helpfulness thresholds are applied.

8. Evaluation Metrics

The task is binary classification, but accuracy alone is not sufficient. Fraud detection is often class-imbalanced, and the cost of false negatives can be high. A model that predicts most users as benign may achieve acceptable accuracy while failing to detect suspicious users. Therefore, the report evaluates multiple metrics.

For this project, F1-score and ROC-AUC are especially important. F1-score is useful when the positive fraud class is imbalanced, while ROC-AUC indicates whether the model ranks suspicious users above benign users across different thresholds. The GNN implementation also uses a probability threshold of 0.35 in evaluation, which increases sensitivity to the fraud class.

9. Results Reporting

The implemented framework generates result files after execution. The feature-based pipeline saves model_table.csv and model_results.csv, while the GNN pipeline saves gnn_results.csv and grey_user_predictions.csv.

Since the dataset and full raw outputs are not included in the submission, no fabricated results are reported. Instead, the following table summarises actual executed results across multiple runs and configurations, including variations in sample size, hidden dimension, learning rate, and the inclusion of LLM explanations.

Model comparison

Key empirical observations

• The best F1-score (0.691) is achieved by the GNN with LLM features.
• The best ROC-AUC (0.877) is achieved by the graph-time logistic regression baseline, indicating that engineered temporal features are highly informative.
• The GNN consistently achieves the best recall (up to 0.875), meaning it is most effective at identifying suspicious users.
• Precision improves when LLM-derived behavioural signals are included, reducing false positives.

10. Discussion

The results reveal a consistent pattern: no single modality dominates across all metrics, and the most informative signals differ depending on what aspect of fraud is being measured.

Temporal and graph-structural features are the strongest primary signals. The graph-time logistic regression model achieves the highest ROC-AUC (0.897 for tfidf_plus_graph_logreg; 0.559 for the purely behavioural baseline), confirming that temporal dynamics — reviewing bursts, recency patterns, account age — and network position encode robust discriminative information. Crucially, these signals are learnable without deep architectures, which implies that much of the fraudulent behaviour in this dataset manifests through structured, detectable patterns rather than subtle semantic cues.

Graph neural networks recover what feature engineering misses. The bipartite GNN achieves the highest recall (0.875), substantially outperforming all feature-based models on this metric. This reflects the architecture's ability to propagate information across the user–product graph: a user's suspiciousness is informed not just by their own activity, but by the products they reviewed and who else reviewed those products. This relational context is precisely what projection-based or bag-of-words approaches discard. The trade-off is lower precision, which reflects both the noisiness of proxy labels and the GNN's tendency to flag borderline cases — a consequence of optimising for recall in an imbalanced setting.

LLM-derived signals act as a precision refinement layer, not a primary driver. Adding LLM-generated behavioural explanations raises precision from 0.568 to 0.613 and improves F1-score from 0.689 to 0.691, while marginally reducing recall. This pattern suggests that language model outputs capture reviewer-level behavioural signatures — stylistic consistency, linguistic implausibility, generic sentiment — that complement graph structure but do not independently identify fraud. Their value is in filtering false positives rather than surfacing new true positives.

The precision–recall trade-off is the central tension. Models with high recall (GNN, graph-time logistic regression) tolerate more false positives; models with high precision (combined random forest, GNN with LLM) are more conservative but miss more actual fraud. The appropriate operating point depends on deployment context: a platform prioritising user trust may accept lower recall to avoid incorrectly flagging legitimate reviewers, while one prioritising review integrity may prefer the high-recall GNN configuration. The F1-score and ROC-AUC results suggest the GNN with LLM explanations offers the best practical balance across both objectives.

Label noise remains the primary confound. Precision variability across runs — and the relatively modest absolute precision values across all models — is best attributed to the use of helpfulness-ratio proxy labels rather than verified fraud annotations. Models are being trained and evaluated against an imperfect signal, which compresses the ceiling on precision regardless of architecture. This does not undermine the comparative findings, but it means absolute metric values should be interpreted cautiously rather than as estimates of real-world performance.

11. Novelty and Significance

The novelty of this work lies in the integration of:

·       temporal behavioural signals

·       graph-based user-product relationships

·       text embeddings (sentence transformers)

·       LLM-generated behavioural interpretations

Across experiments, the strongest performance is achieved when these signals are combined, particularly within the GNN framework. However, classical graph-time models remain highly competitive, reinforcing the importance of well-designed feature engineering in fraud detection.

12. Limitations and Ethical Considerations

The primary limitation of this work is the reliance on proxy labels derived from helpfulness ratios as a surrogate for ground-truth fraud annotations. While pragmatic, this introduces label noise that propagates into training and is reflected in the precision variability observed across models. The absence of verified human-labelled fraudulent reviews constrains the ceiling of supervised performance.

Dataset scale also poses a challenge. Experiments conducted across 1,000–40,000 samples demonstrate broadly consistent trends, but smaller subsets exhibit instability in gradient-based and graph-based models — particularly in recall, where class imbalance effects are amplified at low sample counts.

Finally, while LLM-generated behavioural explanations meaningfully improve interpretability and contribute moderate precision gains, they introduce non-trivial computational overhead. Their utility as scalar signals, rather than richer structured representations, limits the extent to which language model capacity is leveraged within the current pipeline.

13. Future Work

Several directions offer strong potential for extending this framework:

·       Heterogeneous GNN architectures — Modelling user–product–review relationships as a multi-relational graph would enable more expressive message passing and better capture the asymmetric nature of fraudulent coordination.

·       Temporal graph neural networks — Incorporating dynamic edge formation over time would allow the model to detect evolving fraud patterns, such as coordinated review bursts, which static graphs cannot represent.

·       Self-supervised and pseudo-label refinement — Reducing dependence on noisy proxy labels through self-training or contrastive pre-training is among the most impactful near-term directions. Iterative pseudo-label refinement in particular could substantially reduce label noise without requiring additional annotation.

·       Structured LLM integration — Rather than treating LLM outputs as scalar behavioural signals, future work should explore using them as rich feature vectors or as structured reasoning inputs to the GNN, better exploiting the representational capacity of large language models.

Of these directions, self-supervised pseudo-label refinement offers the clearest path to performance improvement within the existing architecture, and is recommended as the immediate next step.

14. Conclusion

This project demonstrates that fraudulent review detection is substantially improved by a multi-view learning framework that integrates temporal, textual, and graph-based signals. No single modality proves sufficient in isolation; the most effective models consistently exploit complementary information across these views.

Among all approaches evaluated, graph-time logistic regression achieves the strongest ROC-AUC, indicating that combining graph-structural and temporal features yields the most reliable discriminative signal. The bipartite GNN delivers the best overall balance of recall and F1-score, demonstrating that relational modelling of user–product interactions captures fraud patterns that feature-based classifiers miss. Incorporating LLM-derived behavioural explanations further improves precision and enhances the interpretability of model outputs.

Collectively, these results confirm that representing Amazon review data as a user–product interaction graph is an effective and principled approach to fraud detection — particularly when temporal dynamics and language-based behavioural signals are incorporated alongside graph structure. The framework laid out here provides a foundation for future work on scalable, label-efficient graph learning for platform integrity.

14. How to Reproduce the Project

Place the downloaded SNAP file in the following path:

data/reviews_Musical_Instruments.json.gz

Install dependencies:

pip install pandas numpy scikit-learn networkx torch sentence-transformers scipy python-louvain

Run the feature-based ML pipeline:

python main.py

Run the GNN pipeline:

python train_gnn.py

The main generated outputs are:

outputs/model_table.csv
outputs/model_results.csv
outputs/gnn_results.csv
outputs/grey_user_predictions.csv

References

He, X., Bresson, X., Laurent, T., Perold, A., LeCun, Y., & Hooi, B. (2024). Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning. International Conference on Learning Representations (ICLR).

Yu, J., Wang, H., Wang, X., Li, Z., Qin, L., Zhang, W., Liao, J., & Zhang, Y. (2023). Group-based Fraud Detection Network on e-Commerce Platforms. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 5463-5475.

SNAP. Amazon product review dataset: Musical Instruments reviews. Stanford Network Analysis Project.

Appendix A. Implementation Map