Detecting Asks in Social Engineering Attacks: Impact of Linguistic and Structural Knowledge
Authors: Bonnie Dorr, Archna Bhatia, Adam Dalton, Brodie Mather, Bryanna Hebenstreit, Sashank Santhanam, Zhuo Cheng, Samira Shaikh, Alan Zemel, Tomek Strzalkowski7675-7682
AAAI 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We describe multiple experiments to determine the impact of linguistic and structural knowledge on ask/framing detection. We present a range of experiments and results and discuss the upshot of our experiments and present related work, contrasting prior approaches to our own. This section describes a range of different experiments to demonstrate the impact of linguistic and structural knowledge on ask/framing detection. |
| Researcher Affiliation | Collaboration | 1Institute for Human and Machine Cognition (IHMC), Ocala, FL, USA, {bdorr, abhatia, adalton, bmather}@ihmc.us 2State University of New York, Albany, NY, USA, {bhebenstreit, azemel}@albany.edu 3University of North Carolina, Charlotte, NC, USA, {sshaikh2, ssantha1, zcheng5}@uncc.edu 4Rensselaer Polytechnic Institute, Troy, NY, USA, tomek@rpi.edu |
| Pseudocode | No | The paper describes algorithmic steps in Section 3.3 but does not present them in a structured pseudocode or algorithm block format. |
| Open Source Code | Yes | By-products of this study are available at a website henceforth referred to as Ask Detection webpage, for a larger project called PANACEA: https://social-threats.github.io/panacea-ask-detection/. |
| Open Datasets | Yes | The resulting validation set is used as a form of ground truth (see Ask Detection webpage) against which we measure clause-level precision/recall/F, as described below. The Ask Detection webpage URL: https://social-threats.github.io/panacea-ask-detection/. |
| Dataset Splits | Yes | We produce a validation set through human adjudication and correction (by a computational linguist) of initial ask/framing labels automatically assigned by our system to SRL-processed clauses from a held-out test set of 20 emails (472 clauses). |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU/CPU models, processor types, or memory amounts used for running experiments. |
| Software Dependencies | No | Detection of the actions associated with asks and framing relies on constituency parses and dependency trees, both taken from Stanford Core NLP (Manning et al. 2014). No version numbers for Stanford Core NLP or other tools are mentioned. |
| Experiment Setup | No | Because our approach uses linguistically-motivated rules coupled with structural knowledge, no training data are needed. (Automatic confidence scoring, training on actual data, is an area of future work.) This indicates no experimental setup with hyperparameters for model training is provided. |