提交 2231e18e 编辑于 作者: Wang JiaJu's avatar Wang JiaJu
浏览文件

Git Init

上级 55bb2ec8
加载中
加载中
加载中
加载中
+138 −57
原始行号 差异行号 差异行
# AIOpsChallenge2025-Submission
# 提交脚本使用指南

本指南介绍如何在竞赛中向评测服务器提交答案。该脚本允许您提交一个答案文件,并在操作成功时获得提交 ID。

## 环境要求

## Getting started
在开始之前,请确保您的系统已安装 Python 3。

To make it easy for you to get started with GitLab, here's a list of recommended next steps.
## 脚本概览

Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
脚本接受一个 JSON Lines 文件(`*.jsonl`),每行是一个单独的 JSON 对象,代表一个问题回答。

## Add your files
## 答案格式

- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
字段说明如下:

| 字段名           | 类型         | 是否必须 | 说明                                                                           |
|----------------|------------|-------------|------------------------------------------------------------------------------|
| component      | string     | 是      | 根据组件的名称,每条样本只评估一个根组件;若提交多个组件,仅评估 JSON 中首个出现的 `component` 字段,类型需为 string。                   |
| reason         | string     | 是      | 故障发生的原因或类型。                                            |
| uuid           | string | 是      | 对应问题的 `uuid`                             |
| reasoning_trace | list   | 是      | 完整推理轨迹,每个推理包含 `step`,`action`,`observation` 三个字段,其中 `observation` 超过 100 字将被截断,仅保留前 100 字参与评分。                         |


**示例**

问题文件:
```
[
    {
        "uuid": "3xxxxx",
        "Anomaly Description": "The system experienced an anomaly from 2025-06-01T16:10:02Z to 2025-06-01T16:31:02Z. Please infer the possible cause."
    },
    {
        "uuid": "4xxxxx",
        "Anomaly Description": "Please analyze the abnormal event between 2025-06-02T17:10:04Z and 2025-06-02T17:33:04Z and provide the root cause."
    }
]
```
提交的答案文件:
```
{"uuid": "3xxxxx", "reason": "high cpu usage", "component": "checkoutservice", "reasoning_trace": [{"step":1, "action":"", "observation":""}]}
{"uuid": "4xxxxx", "reason": "high cpu usage", "component": "checkoutservice", "reasoning_trace": [{"step":1, "action":"", "observation":""}]}
```

注1:答案数量必须和问题保持一致,否则无法成功提交。

注2:答案中的uuid必须与问题的uuid一一对应,否则无法成功提交。

注3:答案的格式必须仅为`uuid`,`reason`,`component`,`reasoning_trace`四个字段,不能增加或缺失字段,类型也必须保持一致,否则无法成功提交。

注4: reasoning_trace 为包含多个 step 对象的数组,每个对象应包含以下字段:
- step:整数,表示推理步骤编号(从 1 开始);
- action:字符串,描述该步调用或操作;
- observation:字符串,描述该步观察到的结果,建议控制在 100 字内;

## 命令行提交

要从命令行使用该脚本,请切换到该脚本所在目录,用 Python 运行该脚本:

```bash
python submit.py [-h] [-s SERVER] [-c CONTEST] [-k TICKET] [result_path]
```
cd existing_repo
git remote add origin http://www.aiops.cn/gitlab/aiops-live-benchmark/aiopschallenge2025-submission.git
git branch -M main
git push -uf origin main

* `[result_path]`:提交的结果文件路径。如果未指定,默认使用当前目录下的 `result.jsonl`
* `-s, --server`:指定评测服务器的 URL。如果未提供,将使用脚本中定义的 `JUDGE_SERVER` 变量。
* `-c, --contest`:比赛标识。如果未提供,将使用脚本中定义的 `CONTEST` 变量。
* `-k, --ticket`:团队标识。如果未提供,将使用脚本中定义的 `TICKET` 变量。
* `-i, --submission_id`:提交标识。如果提供,脚本将查询本次提交的评测状态。

## 编程方式提交

您还可以将 `submit` 函数导入到您的 Python 代码中,以便用编程方式提交数据。


1. 导入函数:
    确保提交脚本位于您的项目目录或 Python 路径中。使用以下方式导入 submit 函数:

    ```python3
    from submit import submit
    ```

## Integrate with your tools
2. 调用 submit 函数:
    准备您的提交数据为字典列表,每个字典代表一个要提交的问题回答。调用 submit 函数:

- [ ] [Set up project integrations](http://www.aiops.cn/gitlab/aiops-live-benchmark/aiopschallenge2025-submission/-/settings/integrations)
    ```python3
    data = [
        {"uuid": "3xxxxx", "reason": "xxx", "component": "", "reasoning_trace": []}
        # 根据需要添加更多项
    ]

## Collaborate with your team
    submission_id = submit(data, judge_server='https://judge.aiops.cn', contest='YOUR_CONTEST_ID', ticket='YOUR_TEAM_TICKET')
    if submission_id:
        print("提交成功!提交 ID: ", submission_id)
    else:
        print("提交失败")
    ```
    
    在此示例中,请将 `YOUR_CONTEST_ID` 替换为您参加的**比赛ID**,将 `YOUR_TEAM_TICKET` 替换为您的**团队ID**。
    *  **比赛ID** 在比赛的URL中获得,比如"赛道一(Qwen1.5-14B):基于检索增强的运维知识问答挑战赛"的URL为https://competition.aiops-challenge.com/home/competition/1771009908746010681 ,比赛ID为1771009908746010681
    *  **团队ID**需要在参加比赛并组队后能获得,具体在比赛详情页-> 团队 -> 团队ID,为一串数字标识。 

3. 调用 check_status 函数
    准备提交ID。调用 check_status 函数:

    ```python3
    submission_id = ""
    status = check_status(submission_id, judge_server='https://judge.aiops.cn', contest='YOUR_CONTEST_ID', ticket='YOUR_TEAM_TICKET')
    if status:
        submission_id = status.get('submission_id')
        score = status.get('score')
        create_time = status.get('create_time')
        judge_time = status.get('judge_time')

        if not judge_time: 
            print("Submission %s is still in queue." % submission_id)
        else:
            print("Submission %s score: %s" % (submission_id, score))
    else:
        print("Failed to check submission status.")
    ```

- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
- [ ] [Set auto-merge](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
## 常见错误

## Test and Deploy
1. 状态码 400,Invalid submission format

Use the built-in continuous integration in GitLab.
    提交的答案文件里,答案的字段和规定不符,请重新检查,须严格满足以下要求:

    ```
    uuid: str
    reason: str
    component: str
    reasoning_trace: list
    ```

- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing (SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
2. 状态码 400,Submission length mismatch

***
    提交的答案文件长度和问题长度不匹配。

# Editing this README
3. 状态码 400,Submission UUID invalid

When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thanks to [makeareadme.com](https://www.makeareadme.com/) for this template.
    提交的答案文件里,答案uuid和问题uuid没有一一对应。

## Suggestions for a good README
4. 状态码 401,Ticket not provided

Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
    没有提供团队ID。

## Name
Choose a self-explaining name for your project.
5. 状态码 401,Invalid ticket

## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
    提供的团队ID不存在。

## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
6. 状态码 401,Contest not provided

## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
    没有提供比赛ID。

## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
7. 状态码 401,Invalid contest

## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
    提供的比赛ID不存在。

## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
8. 状态码 401,Submission ID not provided

## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
    查询提交状态时,没有提供submission ID。

## Contributing
State if you are open to contributions and what your requirements are for accepting them.
9. 状态码 403, Quota exceeded

For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
    提交次数超过限额。

You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
10. 状态码 404,Submission not found

## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
    没有找到对应的提交记录。

## License
For open source projects, say how it is licensed.
11. 状态码 429,Too many requests

## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
    提交过于频繁。

result.jsonl

0 → 100644
+211 −0

添加文件。

预览已超出大小限制,变更已折叠。

submit.py

0 → 100755
+133 −0
原始行号 差异行号 差异行
#!/usr/bin/env python3

import json
import argparse
from urllib import request, error


# 提交答案服务域名或IP
JUDGE_SERVER = "https://judge.aiops.cn"
# 比赛ID,可通过比赛界面 URL 获取, 比如"赛道一(Qwen1.5-14B):基于检索增强的运维知识问答挑战赛"的URL为https://competition.aiops-challenge.com/home/competition/1771009908746010681 ,比赛ID为1771009908746010681
CONTEST = ''
# 团队ID, 需要在参加比赛并组队后能获得,具体在比赛详情页-> 团队 -> 团队ID,为一串数字标识。 
TICKET = ''


def submit(data, judge_server=None, contest=None, ticket=None):
    judge_server = judge_server or JUDGE_SERVER
    contest = contest or CONTEST
    ticket = ticket or TICKET
    
    if not judge_server or not contest or not ticket:
        missing = [
            "judge_server" if not judge_server else "",
            "contest" if not contest else "",
            "ticket" if not ticket else "",
        ]
        missing = [m for m in missing if m]
        print("Required fields must be provided: %s" % ', '.join(missing))
        return None
    
    req_data = json.dumps({'data': data}).encode('utf-8')
    req = request.Request(judge_server, data=req_data, headers={'ticket': ticket, 'contest': contest, 'Content-Type': 'application/json'})

    try:
        with request.urlopen(req) as response:
            response_body = response.read().decode('utf-8')
            submission_id = json.loads(response_body)['submission_id']
            remaining_attempts = json.loads(response_body).get('remaining_attempts', -1)
            return submission_id, remaining_attempts
    except error.HTTPError as e:
        msg = e.reason
        response_body = e.read().decode('utf-8')
        if response_body:
            try:
                msg = json.loads(response_body)['detail']
            except:
                pass
        print("[Error %s] %s" % (e.code, msg))

    except error.URLError as e:
        print(e.reason)
        return None
    
def check_status(submission_id, judge_server=None, contest=None, ticket=None):
    judge_server = judge_server or JUDGE_SERVER
    contest = contest or CONTEST
    ticket = ticket or TICKET
    
    if not judge_server or not contest or not ticket or not submission_id:
        missing = [
            "judge_server" if not judge_server else "",
            "contest" if not contest else "",
            "ticket" if not ticket else "",
            "submission_id" if not submission_id else "",
        ]
        missing = [m for m in missing if m]
        print("Required fields must be provided: %s" % ', '.join(missing))
        return None
    
    req = request.Request(judge_server + "/status", headers={'ticket': ticket, 'contest': contest, 'submission': submission_id, 'Content-Type': 'application/json'})

    try:
        with request.urlopen(req) as response:
            response_body = response.read().decode('utf-8')
            status = json.loads(response_body)
            return status
    except error.HTTPError as e:
        msg = e.reason
        response_body = e.read().decode('utf-8')
        if response_body:
            try:
                msg = json.loads(response_body)['detail']
            except:
                pass
        print("[Error %s] %s" % (e.code, msg))

    except error.URLError as e:
        print(e.reason)
        return None

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Submit to judge server")
    parser.add_argument('result_path', nargs='?', default='result.jsonl', help='Path to the submission file, default is result.jsonl')
    parser.add_argument('-s', '--server', help='Judge server URL, if not specified, the global JUDGE_SERVER variable will be used')
    parser.add_argument('-c', '--contest', help='Contest ID, if not specified, the global CONTEST variable will be used')
    parser.add_argument('-k', '--ticket', help='Submission ticket, if not specified, the global TICKET variable will be used')
    parser.add_argument('-i', '--submission_id', help='Submission ID, specified if you want to check the submission status', default=None)
    
    args = parser.parse_args()

    if args.submission_id:
        status = check_status(args.submission_id, judge_server=args.server, contest=args.contest, ticket=args.ticket)
        if status:
            submission_id = status.get('submission_id')
            score = status.get('score')
            create_time = status.get('create_time')
            judge_time = status.get('judge_time')

            if not judge_time: 
                print("Submission %s is still in queue." % submission_id)
            else:
                print("Submission %s score: %s" % (submission_id, score))
            exit(0)
        else:
            print("Failed to check submission status.")
            exit(1)

    try:
        with open(args.result_path, 'r') as file:
            data = [json.loads(line.strip()) for line in file if line.strip()]
    except Exception as e:
        print(e)
        exit(1)

    return_data = submit(data, judge_server=args.server, contest=args.contest, ticket=args.ticket)
    if return_data:
        submission_id, remaining_attempts = return_data
        print("Success! Your submission ID is %s." % submission_id)
        if remaining_attempts >= 0:
            print("You have %d remaining evaluation attempt(s)." % remaining_attempts)
        exit(0)
    else:
        exit(1)